= Welcome to the dataMine plugin for Vera = '''Note:''' This page is work in progress!!! == Overview of dataMine == ''dataMine'' is a plugin for logging selected UPnP variables to a file, and provides a frontend UI interface to allow graphing of the logged data. The graphing system supports interactive zooming and scrolling to allow the data to be manipulated as the user desires. ''dataMine'' produces 3 history files to support different levels of zooming, and automatically chooses the correct level of data to display depending on the zoom level. When displaying the highest resolution data, the data is displayed down to the second, and when zoomed out, a subsampled dataset is displayed. The UI dynamically interacts with the user and Vera to display the most appropriate data at the time. == Installation == As ''dataMine'' makes use of a number of javascript libraries (primarily the flotr graphing package) installation isn't quite as straight forward as a standard plugin. The usual files (XML, JSON, LUA) are loaded in the normal way through the "developers" button in the control panel and all files specific to ''dataMine'' are placed into a directory /www/cmh/plugins/dataMine - this includes the js files, and a number of icons for buttons. ''dataMine'' can generate a lot of data, so it is recommended to install a USB stick. Click on MiOS developpers Go to the "Luup Files" tab and upload the following files -: - L_DataMine1.lua - D_DataMine1.xml - S_DataMine1.xml - I_DataMine1.xml - D_DataMine1.json - J_DataMine1.js Go to the "Create Device" tab and set the following values-: - '''{{{UpnpDevFilename}}}''' D_DataMine1.xml - '''{{{UpnpImplFilename}}}''' I_DataMine1.xml - '''{{{DeviceType}}}''' urn:cd-jackson-com:serviceId:DataMine1:1 - '''{{{Description}}}''' dataMine Use WinSCP (or another SSH client) to upload the following files to Vera Directory: /www/cmh/plugins/dataMine - All the GIF/PNG/JS files (other than the files listed above) Directory: /www/cmh/skins/default/icons - Data_Mine.png The plugin needs the json library. To install this, go to -: - http://code.mios.com/trac/mios_genericutils/export/20/json.lua and download this to the following directory on Vera - /usr/lib/lua Format a USB stick and mount it to the Vera. (I was unable to get the Vera USB log command to work.) I formatted my drive using FAT32 on my Windows 7 machine and it worked a treat. - md /dataMine - mount /dev/scsi/host0/bus0/target0/lun0/part1 /dataMine Modify the /etc/init.d.custom-user-startup file to add the line -: - mount /dev/scsi/host0/bus0/target0/lun0/part1 /dataMine & This ensures the USB stick is remapped when Vera restarts Once the ''dataMine'' device is running, click on the tool icon. You will get the graph tab, and config tab. Click on the config tab, and select the variables you want to log. All changes to the selected variables should now be logged to files in the /dataMine directory. == Use == The ''dataMine'' plugin installs as a device, and by clicking on the tools button you are presented with two tabs. The first one is the graph tab. This has a listbox to list all the available variables, and a number of buttons. The buttons allow you to select the last day/week/month, or all data, and to scroll and zoom the data. You can also zoom by selecting an area of the graph with the mouse. If you put the mouse close to a data point, the datapoint will be highted with a red circle, and the time and value will be displayed in the bottom righ corner of the graph. (note that there is an issue here - the background UI needs to be scrolled to the top or the mouse position will be incorrect). [[Image(dataMine1.png)]] The config tab allows you to select the variables to log. This provides a list of all variables for devices that aren't hidden. Variables are listed by service, and if you click on the variable name, logging will be enabled. Clicking again will disable logging, but the variable will still be listed in the graph page dropdown. There's also a little clock button for selected variables. Clicking on this will generate the hourly/daily data for the different zoom levels (currently this is only manual, but it will be made automatic soon). [[Image(dataMine2.png)]] == Under the hood == ''dataMine'' records data in CSV files (Comma Separated Value). (Currently) three different files are generated for each variable being recorded - a raw, hourly and daily file. Files are then split to avoid them getting too large (which is a potential problem for constantly changing sensor data such as temperature). Each time Vera observes a change in the data for a monitored variable, dataMine will log the current value into a file. Each day, the long term history files are generated from the raw data. Each line of data in the CSV logfiles includes two numbers - the time of the datapoint (as a unix timestamp), and the data value. For the raw file, the actual changes are logged. For the hourly and daily files, the minimum and maximum value and time are logged for each time period.