netcdf service
Submitted by ellynm on Fri, 04/03/2009 - 07:46
Published in
HI folks-
I installed dapper, and am trying to use netcdf services, to point to EPIC format netCDF files. Unfortunately the netCDF services page in the documentation is missing, so I'm guessing on how to configure dapper.properties.
At the moment I'm getting error messages that suggest it's having problems connecting to MySql, which I don't want it to do anyhow. I have dapper.netcdf_directory set to an address where the files reside, but would pointing to the OpenDAP URI's be a better choice? What do I need to do to tell it not to look for MysQL?
Is there a location for newbie FAQ's that I've missed?
Thanks! Ellyn
netcdf service
Hi Ellyn,
If all you want to do is serve netCDF files via OPeNDAP, then set the dapper.netcdf_directory to point to the directory where your netCDF files live and start up Dapper. You can ignore the MySQL error message. If you point your browser to http://:8080/dods you should then see the OPeNDAP URLs for your files.
However, since you're using EPIC formatted netCDF files, I'm guessing that these files represent in-situ data and that you want to aggregate these files (in the same way as the Argo data files are aggregated at http://dapper.pmel.noaa.gov). In this case, you should (if you haven't already) work through the example on the installation page, and use the same technique to add your own files to Dapper.
- Joe
netcdf service
Hi Joe-
Thanks for the prompt reply.
I tried putting my root opendap directory in the .netcdf_directory field, restarted dapper, and see a list of one subdirectory (there are many) of the data files when I look at localhost:8080/dods. This is a great start, but I'd hoped I could point dchart at all these files for plotting. When I replace /dods with /dchart in the URL, dchart starts but only shows the gridded example, NCAR reanalysis, none of our data.
Our data is all timeseries, so the global view would be useful for selecting files, but simple x-y plots are all we need to plot the data. If you want to view our data holdings, a sample is at: http://stellwagen.er.usgs.gov/cgi-bin/nph-dods/DATAFILES/GLOBEC_GB/ (& there are about 30 other subdirectories at the level of GLOBEC_GB.)
I'm not a java programer, but the error messages sure look like it's having trouble connecting, but it also wants glibc-2.5 or greater for sqlite. Is this critical, even if I'm now using the database?
An excerpt from the screen dump regarding connection issues follows:
=======================Seeding DChart cache====================
Adding uri http://localhost:8080/dods/dapperexample/argo_sample.cdp md5 = 4f88d158ef8ca59fd39c3c68ea192a
DClientPlotDataset disableCache = false
New url:http://localhost:8080/dods/dapperexample/argo_sample.cdp
Dataset is profile
OPeNDAP proxying disabled = false
Adding database dapperexample
Connecting to database with JDBC URL jdbc:mysql://ellmondtlx.whsc.er.usgs.gov/dapperexample?autoReconnect=true
java.sql.SQLException: Server connection failure during transaction. Due to underlying exception: 'java.net.SocketException: java.net.ConnectException: Connection refused'.
** BEGIN NESTED EXCEPTION **
java.net.SocketException
MESSAGE: java.net.ConnectException: Connection refused
STACKTRACE:
java.net.SocketException: java.net.ConnectException: Connection refused
at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:156)
at com.mysql.jdbc.MysqlIO.(MysqlIO.java:283)
at com.mysql.jdbc.Connection.createNewIO(Connection.java:2666)
I appreciate any additional help-
Ellyn
Hi Ellyn, It looks like MySQL
Hi Ellyn,
It looks like MySQL isn't running on ellmondtlx.whsc.er.usgs.gov. You can ignore the sqlite error message.
You need to aggregate the files using the dapperload.sh command. Once you've aggregated the files, you need to edit the dchart/etc/config.xml file to point to the URL that was created when you aggregated the files.
Here's an example, using the files from the URL you sent me. Assume the files are in a directory named /data/woodshole, and you want to aggregate all of the 'mca' files (note: since your netCDF files contain different variables, you don't want to aggregate them into one dataset).
1) Create the database and dataset:
~/dapper/dapperload.sh -u -p create woodshole mca_time
2) Load the data files:
~/dapper/dapperload.sh -u -p load woodshole mca_time /data/woodshole/*mca*.nc
3) Add the following to dchart/etc/config.xml:
<dataset type="time" category="WHOI" url="http://localhost:8080/dods/woodshole/mca_time.cdp" name="WHOI Stuff">
<attributes>
<stationName variable="MOORING"/>
</attributes>
</dataset>
If you point your browser to localhost:8080/dchart, you should be able to plot your data (or visualize the data in Google Earth).
SQlite
Hi-
Are you saying that in order to use Dchart, the user MUST aggregate? I'd like to just use it to plot the data in the existing files.
You're right, I don't have MySql running, but you said earlier that it wasn't necessary. I tried the dapperload examples from your previous email (ignoring error messages), and .xml files are created in resources but I couldn't find any .cdp file. Should one be created locally? When I modify the config.xml as suggested, dchart starts, but does not show any files or variables (because there's no .cdp to open?).
Does Dchart ONLY work with data served by dapper and aggregated by dapperload to create a .cdp file? I tried pointing to a .nc file on our OpenDAP server, and that didn't do the trick.
The problem may be that our files don't conform to Dchart's sequence requirements- we have no id variable, latitude is called lat, and longitude is called lon, and we employ the older time and time2 way of expressing "time". Our depth variable is called depth, so one thing matches.
Please let me know what I'm missing.
Thanks- Ellyn
Hi Ellyn, There seems to be a
Hi Ellyn,
There seems to be a bit of confusion here. First off, Dapper is an OPeNDAP server. DChart is a server that provides visualization and download access to various kinds of data, including data served by Dapper.
Dapper serves two kinds of data using the OPeNDAP network protocol -- gridded data (as in data generated from models), and in-situ or station data. Typically (as in your case), in-situ data consists of hundreds or thousands of very small netCDF files. Now, Dapper can be configured to serve each of these files as a separate dataset, but this isn't very useful since, in general, there's no way a user can conduct space or time based searches of this data. If DChart were configured to use this data, a user would be presented with hundreds of obscurely named datasets and would have no idea what any of these datasets contain.
So you have to aggregate in-situ data. And if you aggregate in-situ data, you have to run MySQL (my previous comment that it isn't required only applies to the case where you're serving each file as a separate dataset without aggregation). And, of course, if you are using the in-situ aggregation features of Dapper that require MySQL, you no longer can ignore any MySQL error messages.
I'm not sure if you ever managed to get the Argo example described in the documentation to work. If not, I'd suggest working through this example (after configuring MySQL, of course) before attempting to add your own data. BTW, I managed to add your data to my test server, so I know that Dapper can read your data.
clarification
Hi-
Thanks for the clarification- that information needs to go in the main part of the documentation :>
I was hoping that Dchart would help access and display all those pesky in-situ timeseries. There really isn't enough spacial or temporal continuity in the data to allow aggregation in any meaningful way, so Dchart probably isn't the best solution for us. I installed it to test and evaluate, so I got an answer.
Thanks for your help!