HOW TO: Make Udoo Neo an IoT Gateway of Sensors

Discussion in 'UDOO NEO' started by waltervl, Nov 13, 2016.

  1. waltervl

    waltervl Well-Known Member

    Joined:
    Dec 12, 2015
    Messages:
    824
    Likes Received:
    254
    Introduction
    This HOW TO is the first of two. This one meant for beginners who just bought a Neo and want to connect it to the Internet and want to see their sensor data on the internet. The second will be about controlling the Neo Arduino in- and ouputs through a local website on your Neo so you can control it from everywhere in your house and from every connected device (smartphone, laptop, desktop) that has a web browser.
    Thingspeak.JPG
    So this HOW TO is about sending sensor data to an IoT dashboard cloud supplier like Thingspeak. But there are more like for example data.sparkfun.com

    Summary
    1: Create a Thingspeak channel so we can upload data to this IoT cloud hosting.
    2: Connect the Neo to the internet so it can connect to Thingspeak
    3: Create a Python program on the Linux side to send the data to Thingspeak.
    4: Create an Arduino Sketch for reading sensor data and sending it to the Linux side
    5. Check your Thinspeak chanel and Confirm data transfer
    6. Read multiple Sensors and send to Thingspeak

    Let's begin! If you have any questions on this How To you are free to ask, just comment in this thread.

    Step 1: Create a Thingspeak channel
    Create Thingspeak channel with 1 temp field. You can use the following instruction: https://nl.mathworks.com/help/thingspeak/collect-data-in-a-new-channel.html

    Step 2:
    Connect the Neo to the internet.
    For connecting the Neo to the internet you can use the following instruction: Use Neo as a headless IoT Device

    Step 3
    : Create a Python program on the Linux side to send the data to Thingspeak.
    Copy this Python code, save it as UdooNeoSensor.py in your home folder on the Neo.
    Replace the key value with your own Thingspeak Write API Key
    Python is preinstalled on the Neo so you dont have to install anything for this.
    After saving this python code you can run it ín a linux terminal console: sudo python UdooNeoSensor.py
    We are using the Linux /dev/ttyMCC serial device that is implemented on the Neo for communicating between the Linux and Arduino side.
    Tip: To copy the code open the Neo desktop through VNC Remote Desktop so you can copy/paste it from your web browser on your PC to the text editor (leafpad) on the Neo with the standard Ctrl-C Ctrl-V commands..
    Code:
    #!/usr/bin/env python
    # This program logs a Arduino Sensor on an Udoo Neo to a Thingspeak Channel
    # To use, get a Thingspeak.com account, set up a channel, and capture the Channel Key at https://thingspeak.com/docs/tutorials/
    # Then paste your channel ID in the code for the value of "key" below.
    # Then run as sudo python UdooNeoSensor.py
    
    import httplib, urllib
    import time
    import serial
    sleep = 16 # how many seconds to sleep between posts to the channel, API limit min 15 sec
    key = 'Put your Thingspeak Channel Key here'  # Thingspeak channel to update
    
    DEBUG = False # set to True to  get more debug prints in the Python console
    
    #Create serial device for reading serial
    arduino = serial.Serial('/dev/ttyMCC',115200,timeout=0)
    arduino.flushOutput()
    arduino.flushInput()
    serial_buffer = ""
    line = ""
    
    temp_str = '0'
    temp = 0
    
    #init for arduino so Arduino knows it can send data.
    arduino.write("I\n")
    
    def ReadArduino(arduino_out):
        global serial_buffer, line
        arduino_out = ""
        serial_data = arduino.read()                 # Read 1 character
        while serial_data:                             # If there is a character proceed
            serial_buffer += serial_data            # Add character to buffer
            if DEBUG: print "ReadArduino:Serial buffer:",(repr(serial_buffer))
            # if there is a new line in it, then create Line string
            if "\r\n" in serial_buffer:
                if DEBUG: print "ReadArduino:New Line:",(repr(serial_buffer))
                line = serial_buffer
                serial_buffer = "" # empty Serial buffer
            serial_data = arduino.read()
            # if there is a Line string then finish reading the Line
            if line:
                    # strip string for unwanted characters
                    line = line.replace("\r\n", "")
                    arduino_out = line
                    if DEBUG: print "ReadArduino:arduino_out:", arduino_out
                    line = ""
                    if arduino_out == "": arduino_out = None
                    # add the last character to the new buffer
                    serial_buffer += serial_data
                    return arduino_out
    
    #Report Udoo Neo Arduino Sensor data to Thingspeak Channel
    def writesensordata():
        global temp, temp_str
        while True:
            #read arduino data and make it a integer value
            temp_str = ReadArduino(temp_str)
            if temp_str is None:
                temp_str = '0'
            temp = int(temp_str)
            params = urllib.urlencode({'field1': temp, 'key':key })
            headers = {"Content-typZZe": "application/x-www-form-urlencoded","Accept": "text/plain"}
            conn = httplib.HTTPConnection("api.thingspeak.com:80")
            try:
                conn.request("POST", "/update", params, headers)
                response = conn.getresponse()
                print temp
                print response.status, response.reason
                data = response.read()
                conn.close()
            except:
                print "connection failed"
            break
    #sleep for desired amount of time
    if __name__ == "__main__":
            while True:
                    writesensordata()
                    time.sleep(sleep)
    Step 4: Create Arduino Sketch for reading sensor end sending it to Serial
    Use the following code to read an arduino analog (temperature) sensor on pin A0 and send it to serial (see also https://www.arduino.cc/en/Tutorial/AnalogReadSerial for more info). I modified the code a bit to slow down the read frequency and to be sure the data is send to the Linux side of the Neo.
    Copy the code in a new sketch (as a start use the IDE on the Neo) and save it. Compile and upload it to the Arduino side of the Neo.
    You can check the results of this sketch with the Serial Monitor of the Arduino IDE.
    You first have to send a character to the Arduino in order to get the sensor data displayed (every 16 seconds). Be sure the Python program is not running else you will get a reading conflict of the /dev/ttytMCC serial device.
    Code:
    void setup() {
      // initialize serial communication:
      Serial.begin(115200);
      delay(100);
      pinMode(13, OUTPUT);
    }
    
    // the loop routine runs over and over again forever:
    void loop() {
      while ( Serial.available() > 0 ) {
      // read the input on analog pin 0:
      int sensorValue = analogRead(A0);
      // print out the value you read:
      digitalWrite(13, HIGH);  // turn the LED on (HIGH is the voltage level)
      delay(1000);  // wait for a second.
      Serial.println(sensorValue); // If LED stays high longer you have serial timeout
      digitalWrite(13, LOW);  // turn the LED off by making the voltage LOW
      delay(10000);  // wait for a 10 seconds to prevent buffer overflow
      }
    }
    
    Step 5: Check your Thinspeak channel and Confirm data transfer
    Be sure the python program and the arduino sketch are running.
    Open your Thingspeak channel and you should see the temperature uploading into thinspeak website.

    You can check my data here: https://thingspeak.com/channels/144274
    I did not change the value of the sensor so this is a very straight line :)
    You can download the values and there is an update every 16 seconds:
    2016-08-10 23:22:00 UTC,12,685
    2016-08-10 23:22:17 UTC,13,684
    2016-08-10 23:22:33 UTC,14,685
    2016-08-10 23:22:49 UTC,15,684
    2016-08-10 23:23:06 UTC,16,684
    2016-08-10 23:23:22 UTC,17,685

    Step 6: Read multiple Sensors and send to Thingspeak
    If you have more sensors you can send all data in one string (use Serial.print() and end with a Serial.println() ) to the A9 and split this string into multiple values again in Python. See an example in the comments. http://www.udoo.org/forum/index.php?posts/24362

    Or you could use a sensor ID with the value.
    The Arduino sketch and Python program in the How To: control Arduino from webpage is an example for using multiple sensors.

    References
    Use Neo as a headless IoT Device
    UDOO NEO Serial Communication Examples
    Python programming for beginners

    Disclaimer: I am not a programmer, just a computer enthousiast. The code provide is tested but is not guaranteed to be the best in class. :) The code and How To is written for beginners to give a start in using the Neo. At the moment this How To is work in progress so updates are foreseen.
    [/URL]
     
    Last edited: Jan 5, 2017
    maurizio, modjo, Andrea Rovai and 2 others like this.
  2. waltervl

    waltervl Well-Known Member

    Joined:
    Dec 12, 2015
    Messages:
    824
    Likes Received:
    254
    Andrea Rovai likes this.
  3. Gla

    Gla New Member

    Joined:
    Oct 25, 2016
    Messages:
    25
    Likes Received:
    4
    thanks for your instructions and guidelines waltervl !!!! i have developed a Light Sensor scenario which works perfectly with ThingSpeak!
     
    waltervl and Andrea Rovai like this.
  4. Gla

    Gla New Member

    Joined:
    Oct 25, 2016
    Messages:
    25
    Likes Received:
    4
    Has anyone tried to update 2 or more data fields in the Thingspeak channel?
     
  5. waltervl

    waltervl Well-Known Member

    Joined:
    Dec 12, 2015
    Messages:
    824
    Likes Received:
    254
    I did not but it should not be that difficult.
    To write 2 values (ie temp and value2) change in the python script:
    params = urllib.urlencode({'field1': temp, 'key':key })
    to
    params = urllib.urlencode({'field1': temp, 'field2': value2, 'key':key })

    To get two or more values out of the arduino part of the Neo go to the other How to: Control Arduino from Web page
     
  6. Andriy

    Andriy Member

    Joined:
    Nov 10, 2015
    Messages:
    30
    Likes Received:
    2
    Thanks! It's cool.
    How can I send date from Bricks directly?
     
  7. Andriy

    Andriy Member

    Joined:
    Nov 10, 2015
    Messages:
    30
    Likes Received:
    2
    It's my code

    Code:
    # A library (example) to get the current values for the outboard sensors such as
    # Temperature sensor
    # Barometer
    import httplib, urllib
    import time
    import serial
    sleep = 16 # how many seconds to sleep between posts to the channel, API limit min 15 sec
    key = '99QLS10HUFWWMYIR'  # Thingspeak channel to update
    
    from neo import Temp #import libraries
    from neo import Barometer
    from time import sleep # import for delays
    
    temp = Temp() # init objects p.s. I auto initialize/reset the modules on these calls
    baro = Barometer()
    key = '99QLS1********'  # Thingspeak channel to update
    
    try:
       conn = httplib.HTTPConnection("api.thingspeak.com:80")
    
    except:
        print("Couldn't connect to the API, check your Internet connection")
        sys.exit(0)
    
    #Report Udoo Neo Arduino Sensor data to Thingspeak Channel
    def writesensordata():
        global temp, temp_str
    while True:
            #read arduino data and make it a integer value
            temp_str = baro.getTemp("c") # slower than temp sensor but still works same as temp sensor replace c with f for different modes
            if temp_str is None:
                temp_str = '0'
            temp = int(temp_str)
            params = urllib.urlencode({'field1': temp, 'key':key })
            headers = {"Content-typZZe": "application/x-www-form-urlencoded","Accept": "text/plain"}
            conn.request("POST", "/update", params, headers)
            response = conn.getresponse()
            print temp
            print response.status, response.reason
            data = response.read()
            conn.close()
    #print "" # newline
            sleep(5) # wait a (5) second
     
  8. waltervl

    waltervl Well-Known Member

    Joined:
    Dec 12, 2015
    Messages:
    824
    Likes Received:
    254
    And what is not working? What errors do you get?
     
  9. Andriy

    Andriy Member

    Joined:
    Nov 10, 2015
    Messages:
    30
    Likes Received:
    2
    Everything is working
    But I see echo: I/O error
     
  10. waltervl

    waltervl Well-Known Member

    Joined:
    Dec 12, 2015
    Messages:
    824
    Likes Received:
    254
    Where is that I/O error coming from? Reading the sensors?
    Are you able to print the sensor values to the console so to be sure you can read the brick sensors.
    You could program more print statements so you can debug.
     
  11. Andriy

    Andriy Member

    Joined:
    Nov 10, 2015
    Messages:
    30
    Likes Received:
    2
    I can read sensors and send of the data on thingspeak. But first string i/o error. After this all ok.
    Code:
    # A library (example) to get the current values for the outboard sensors such as
    # Temperature sensor
    # Barometer
    import httplib, urllib
    import time
    import serial
    sleep = 16 # how many seconds to sleep between posts to the channel, API limit min 15 sec
    key = '99QLS10HUFWWMYIR'  # Thingspeak channel to update
    
    from neo import Temp #import libraries
    from neo import Barometer
    from time import sleep # import for delays
    
    temp = Temp() # init objects p.s. I auto initialize/reset the modules on these calls
    baro = Barometer()
    key = '99QLS****'  # Thingspeak channel to update
    
    try:
       conn = httplib.HTTPConnection("api.thingspeak.com:80")
    
    except:
        print("Couldn't connect to the API, check your Internet connection")
        sys.exit(0)
    
    #Report Udoo Neo Arduino Sensor data to Thingspeak Channel
    def writesensordata():
        global temp, temp_str
    while True:
            #read arduino data and make it a integer value
            temp_str = baro.getTemp("c") # slower than temp sensor but still works same as temp sensor replace c with f for different modes
            if temp_str is None:
                temp_str = '0'
            temp = int(temp_str)
            bar_str = baro.getPressure() # gets the pressure in kPA
            barHg=7.50063*bar_str
            if bar_str is None:
                bar_str = '0'
            bar = int( barHg)
            params = urllib.urlencode({'field1': temp, 'field2': bar, 'key':key })
            headers = {"Content-typZZe": "application/x-www-form-urlencoded","Accept": "text/plain"}
            conn.request("POST", "/update", params, headers)
            response = conn.getresponse()
            print temp
            print bar
            print response.status, response.reason
            data = response.read()
            conn.close()
    #print "" # newline
            sleep(1) # wait a (5) second
     
    Last edited: Jan 2, 2017
  12. waltervl

    waltervl Well-Known Member

    Joined:
    Dec 12, 2015
    Messages:
    824
    Likes Received:
    254
    Ok, then all OK.
    I would not send data every second but do it every minute or so. I also have read somewhere that thingspeak only accepts data every 15 seconds.
     
  13. Andriy

    Andriy Member

    Joined:
    Nov 10, 2015
    Messages:
    30
    Likes Received:
    2
    Thanks.
    How can I autoruning python script after starting of ubuntu.
     
  14. Maurice

    Maurice Active Member

    Joined:
    Oct 13, 2015
    Messages:
    229
    Likes Received:
    44
    You could start your script at the end of /etc/rc.local. 'Better' would be to create a Linux service.
     
  15. waltervl

    waltervl Well-Known Member

    Joined:
    Dec 12, 2015
    Messages:
    824
    Likes Received:
    254
    This would be the instruction to do so from link:
    Put this file in /etc/init
    mystartupscript.conf
    Code:
    start on runlevel [2345]
    stop on runlevel [!2345]
    
    exec /path/to/script.py
    By placing this conf file there you hook into ubuntu's upstart service that runs services on startup.
    manual starting/stopping is done with "sudo service mystartupscript start" and "sudo service mystartupscript stop"
     
  16. Andriy

    Andriy Member

    Joined:
    Nov 10, 2015
    Messages:
    30
    Likes Received:
    2
    Does not work :(

    /etc/init/mystartupscript.conf
    ------
    start on runlevel [2345]
    stop on runlevel [!2345]

    exec /home/udooer/Desktop/udoo_thing_job.py
     
  17. waltervl

    waltervl Well-Known Member

    Joined:
    Dec 12, 2015
    Messages:
    824
    Likes Received:
    254
    What does the logfile /var/log/upstart/[servicename] say?
    And perhaps you have to make the .py file executable. or use
    Code:
    exec python /path/to/your/script.py
     
  18. Andriy

    Andriy Member

    Joined:
    Nov 10, 2015
    Messages:
    30
    Likes Received:
    2
    logfile
    Code:
    sh: echo: I/O error
    
    Traceback (most recent call last):
    
      File "/home/udooer/Desktop/udoo_thing_job.py", line 42, in <module>
    
        conn.request("POST", "/update", params, headers)
    
      File "/usr/lib/python2.7/httplib.py", line 1017, in request
    
        self._send_request(method, url, body, headers)
    
      File "/usr/lib/python2.7/httplib.py", line 1051, in _send_request
    
        self.endheaders(body)
    
      File "/usr/lib/python2.7/httplib.py", line 1013, in endheaders
    
        self._send_output(message_body)
    
      File "/usr/lib/python2.7/httplib.py", line 864, in _send_output
    
        self.send(msg)
    
      File "/usr/lib/python2.7/httplib.py", line 826, in send
    
        self.connect()
    
      File "/usr/lib/python2.7/httplib.py", line 807, in connect
    
        self.timeout, self.source_address)
    
      File "/usr/lib/python2.7/socket.py", line 553, in create_connection
    
        for res in getaddrinfo(host, port, 0, SOCK_STREAM):
    
    socket.gaierror: [Errno -2] Name or service not known
    
    
     
  19. waltervl

    waltervl Well-Known Member

    Joined:
    Dec 12, 2015
    Messages:
    824
    Likes Received:
    254
    Looks like the same error as before where you said everything works. Does it really work? Do you see data coming in on your Thingspeak channel with the normal program?
     
  20. Andriy

    Andriy Member

    Joined:
    Nov 10, 2015
    Messages:
    30
    Likes Received:
    2
    After run udoo_thing_job.py
    I see
    Terminal:
    sh: echo: I/O error
    17
    729
    200 OK
    .....
    on ThingSpeak I see
    https://thingspeak.com/channels/138029
    -------
    ThingSpeak script works from terminal but don't works from "sudo service mystartupscript start"
    Ubidots script works. I run script through "sudo service mystartupscript start". But automatically don't start. I must run "sudo service mystartupscript start" through terminal.

    Ubidots code:
    Code:
    #!/usr/bin/python
    # A library (example) to get the current values for the outboard sensors such as
    # Temperature sensor
    # Barometer
    
    from neo import Temp #import libraries
    from neo import Barometer
    
    from time import sleep # import for delays
    
    temp = Temp() # init objects p.s. I auto initialize/reset the modules on these calls
    baro = Barometer()
    
    # Ubidots send
    import time
    import sys
    import commands
    import subprocess
    from ubidots import ApiClient
    
    try:
    
        api = ApiClient(token="UVVUNvgF76upGETU3v9QcHgLR0WMDQ")
    
    except:
    
        print("Couldn't connect to the API, check your Internet connection")
        sys.exit(0)
    
    while True:
        tempVal = temp.getTemp("f") # replace f with c to get celcius
        #print "Current temp from sensor 1: "+str(tempVal) #need to turn into string before building strings
           #Temperature sensor - tempVal
            api.save_collection([{'variable':'5777bbe67625427a308c0c91','value':tempVal}])
    
        pressureVal = baro.getPressure() # gets the pressure in kPA
            pressureValinHg=7.50063*pressureVal
        #print "Current pressure in (kPa):  "+str(pressureVal)
           #Barometer - pressureVal
            api.save_collection([{'variable':'5779289a76254218a67eacaf','value':pressureValinHg}])
    
        tempFromBaro = baro.getTemp("c") # slower than temp sensor but still works same as temp sensor replace c with f for different modes
        #print "Current temp from sensor 2: "+str(tempFromBaro)
          #TempFromBaro
            api.save_collection([{'variable':'5777bbee7625427a763c7332','value':tempFromBaro}])
    
        #print "" # newline
        sleep(1800) # wait a (5) second
    
    ----------------------------------------------------

    ThingSpeak code:
    Code:
    #!/usr/bin/python
    # A library (example) to get the current values for the outboard sensors such as
    # Temperature sensor
    # Barometer
    import urllib
    import httplib
    import urlparse
    import time
    
    sleep = 16 # how many seconds to sleep between posts to the channel, API limit min 15 sec
    key = '99QLS10HUFWWMYIR'  # Thingspeak channel to update
    
    from neo import Temp #import libraries
    from neo import Barometer
    from time import sleep # import for delays
    
    temp = Temp() # init objects p.s. I auto initialize/reset the modules on these calls
    baro = Barometer()
    key = '99QLS10HUFWWMYIR'  # Thingspeak channel to update
    
    try:
       conn = httplib.HTTPConnection("api.thingspeak.com:80")
    
    except:
        print("Couldn't connect to the API, check your Internet connection")
        sys.exit(0)
    
    #Report Udoo Neo Arduino Sensor data to Thingspeak Channel
    def writesensordata():
        global temp, temp_str
    while True:
            #read arduino data and make it a integer value
            temp_str = baro.getTemp("c") # slower than temp sensor but still works same as temp sensor replace c with f for different modes
            if temp_str is None:
                temp_str = '0'
            temp = int(temp_str)
            bar_str = baro.getPressure() # gets the pressure in kPA
            barHg=7.50063*bar_str
            if bar_str is None:
                bar_str = '0'
            bar = int( barHg)
            params = urllib.urlencode({'field1': temp, 'field2': bar, 'key':key })
            headers = {"Content-typZZe": "application/x-www-form-urlencoded","Accept": "text/plain"}
            conn.request("POST", "/update", params, headers)
            response = conn.getresponse()
            print temp
            print bar
            print response.status, response.reason
            data = response.read()
            conn.close()
    #print "" # newline
            sleep(1800) # wait a (5) second
     

Share This Page