Computing/Getting ERDDAP Running - Revision history

Cricalix at 06:28, 13 October 2023

2023-10-13T06:28:52Z

← Older revision		Revision as of 06:28, 13 October 2023
Line 11:		Line 11:
	# Read through https://github.com/unidata/tomcat-docker to see if any Tomcat bits apply (unlikely, will use Caddy2 to front this Tomcat)		# Read through https://github.com/unidata/tomcat-docker to see if any Tomcat bits apply (unlikely, will use Caddy2 to front this Tomcat)
	# Read through https://registry.hub.docker.com/r/axiom/docker-erddap/ and (source) https://github.com/axiom-data-science/docker-erddap		# Read through https://registry.hub.docker.com/r/axiom/docker-erddap/ and (source) https://github.com/axiom-data-science/docker-erddap
	# Run the container at least once and extract /usr/local/tomcat/content/erddap/setup.xml and datasets.xml		# Run the container at least once and extract <code>/usr/local/tomcat/content/erddap/setup.xml</code> and <code>datasets.xml</code>
	# Tune the XML files		# Tune the XML files
	## Even though Axiom's docker can do everything with environment variables, setup.xml must exist		## Even though Axiom's docker can do everything with environment variables, <code>setup.xml</code> must exist
	# Re-run the docker instance with file pass-throughs		# Re-run the docker instance with file pass-throughs
	## Don't forget to pass through the bigParentDirectory, because the logs are there		## Don't forget to pass through the <code>bigParentDirectory</code>, because the logs are there
	## In fact, there's a ton of stuff there that is needed for persistent operation		## In fact, there's a ton of stuff there that is needed for persistent operation
	# A sample JSON file is needed where the dataset indicates the files will exist		# A sample JSON file is needed where the dataset indicates the files will exist
Line 21:		Line 21:
	# Use the DasDds.sh script in the WEB-INF directory to validate the dataset.xml file		# Use the DasDds.sh script in the WEB-INF directory to validate the dataset.xml file
	## Given this is docker, either <code>docker exec -it <name> /bin/bash</code> or invoke it with sufficient pass-throughs to run the script directly from the host		## Given this is docker, either <code>docker exec -it <name> /bin/bash</code> or invoke it with sufficient pass-throughs to run the script directly from the host
			On restart, the service shows the default landing page, but shortly thereafter replaces it with any modified content. This implies that the pass-through of the XML files somehow doesn't take effect until the service has re-read them after docker startup.

	=== Logs ===		=== Logs ===
Line 37:		Line 38:
	# Yes, it's HTTP GET -> data write happens. It's terrible and goes against the RFC for web server implementation.		# Yes, it's HTTP GET -> data write happens. It's terrible and goes against the RFC for web server implementation.

	=== AIS ~~Catcher~~ integration ===		=== AIS -catcher integration ===
	The whole goal of this setup is to grab AIS data from the meteorological sensors in Dublin Bay. A simple proof-of-concept parser follows.		The whole goal of this setup is to grab AIS data from the meteorological sensors in Dublin Bay. A simple proof-of-concept parser follows.

Line 80:		Line 81:
	with urllib.request.urlopen(url) as f:		with urllib.request.urlopen(url) as f:
	print(f.read().decode('utf-8'))		print(f.read().decode('utf-8'))
	</syntaxhighlight>		</syntaxhighlight>A more thorough implementation is found at https://github.com/cricalix/erddap-feeder - a small Rust program that can listen for HTTP POST requests from ais-catcher and transform them to HTTP GET requests against the ERDDAP instance.

Cricalix at 07:20, 25 September 2023

2023-09-25T07:20:49Z

← Older revision		Revision as of 07:20, 25 September 2023
Line 19:		Line 19:
	# A sample JSON file is needed where the dataset indicates the files will exist		# A sample JSON file is needed where the dataset indicates the files will exist
	## The record should be deletable once the system is running and has real data flowing		## The record should be deletable once the system is running and has real data flowing
			# Use the DasDds.sh script in the WEB-INF directory to validate the dataset.xml file
			## Given this is docker, either <code>docker exec -it <name> /bin/bash</code> or invoke it with sufficient pass-throughs to run the script directly from the host

	=== Logs ===		=== Logs ===

Cricalix: /* Configuration */

2023-09-25T07:08:57Z

Configuration

← Older revision		Revision as of 07:08, 25 September 2023
Line 17:		Line 17:
	## Don't forget to pass through the bigParentDirectory, because the logs are there		## Don't forget to pass through the bigParentDirectory, because the logs are there
	## In fact, there's a ton of stuff there that is needed for persistent operation		## In fact, there's a ton of stuff there that is needed for persistent operation
			# A sample JSON file is needed where the dataset indicates the files will exist
			## The record should be deletable once the system is running and has real data flowing

	=== Logs ===		=== Logs ===

Cricalix: Created page with "ERDDAP is software for serving up scientific data. To run and configure it, the Docker approach works fairly well (even running on a Synology NAS). === Installation === # `docker pull axiom/docker-erddap:latest-jdk17-openjdk` === Configuration === # Read https://coastwatch.pfeg.noaa.gov/erddap/download/setupDatasetsXml.html very carefully. ## It's long. It's thorough. # Read through https://github.com/unidata/tomcat-docker to see if any Tomcat bits apply (..."

2023-09-25T07:07:14Z

Created page with "ERDDAP is software for serving up scientific data. To run and configure it, the Docker approach works fairly well (even running on a Synology NAS). === Installation === # <code>docker pull axiom/docker-erddap:latest-jdk17-openjdk</code> === Configuration === # Read https://coastwatch.pfeg.noaa.gov/erddap/download/setupDatasetsXml.html very carefully. ## It's long. It's thorough. # Read through https://github.com/unidata/tomcat-docker to see if any Tomcat bits apply (..."

New page

ERDDAP is software for serving up scientific data. To run and configure it, the Docker approach works fairly well (even running on a Synology NAS).

=== Installation ===

# <code>docker pull axiom/docker-erddap:latest-jdk17-openjdk</code>

=== Configuration ===

# Read https://coastwatch.pfeg.noaa.gov/erddap/download/setupDatasetsXml.html very carefully.
## It's long. It's thorough.
# Read through https://github.com/unidata/tomcat-docker to see if any Tomcat bits apply (unlikely, will use Caddy2 to front this Tomcat)
# Read through https://registry.hub.docker.com/r/axiom/docker-erddap/ and (source) https://github.com/axiom-data-science/docker-erddap
# Run the container at least once and extract /usr/local/tomcat/content/erddap/setup.xml and datasets.xml
# Tune the XML files
## Even though Axiom's docker can do everything with environment variables, setup.xml must exist
# Re-run the docker instance with file pass-throughs
## Don't forget to pass through the bigParentDirectory, because the logs are there
## In fact, there's a ton of stuff there that is needed for persistent operation

=== Logs ===

# Catalina logs are uninteresting
# Application logs are not exported to docker's log system
# Application logs are in the bigParentDirectory passed-through volume

=== HTTPGet errors ===

# <code>sourceName</code> does not do case translations to <code>destinationName</code>. They must be '''identical'''.
# Configuration of the dataset attributes uses (effectively) the <code>destinationName</code> spelling
# <code>time, longitude, latitude</code> must be spelt that way
# <code>author</code> must be the last field on the request, and it points to the keys (username_$password)
# <code>command</code> must exist in the definition, but never set
# Yes, it's HTTP GET -> data write happens. It's terrible and goes against the RFC for web server implementation.

=== AIS Catcher integration ===
The whole goal of this setup is to grab AIS data from the meteorological sensors in Dublin Bay. A simple proof-of-concept parser follows.

<syntaxhighlight lang="python" line="">
#!/usr/bin/env python3

import json
import urllib.request
import urllib.parse
from datetime import datetime

URL="https://erddap.home.arpa/erddap/tabledap/datasetName.insert"
MAPPER = {
"signalpower": "Signal_Power",
"wspeed": "Wind_Speed",
"wgust": "Wind_Gust_Speed",
"wdir": "Wind_Direction",
"wgustdir": "Wind_Gust_Direction",
"lon": "longitude",
"lat": "latitude",
"waveheight": "Wave_Height",
"waveperiod": "Wave_Period",
"mmsi": "MMSI",
}

with open("ais.capture", "r") as f:
lines = f.readlines()

lines = [json.loads(x) for x in lines if "wspeed" in x and "01301" in x]

for line in lines:
kv = {}
rxtime = line["rxtime"]
dt_rxtime = datetime.strptime(rxtime, "%Y%m%d%H%M%S")
kv["time"]=dt_rxtime.strftime('%Y-%m-%dT%H:%M:%SZ')
kv["Station_ID"] ="Dublin_Bay_Buoy"
for field, value in MAPPER.items():
kv[value]=line[field]
kv["author"] ="testuser_somepassword"
params = urllib.parse.urlencode(kv)
url = f"{URL}?{params}"
with urllib.request.urlopen(url) as f:
print(f.read().decode('utf-8'))
</syntaxhighlight>