Recent Met data -- the status and to-dos.

Aug 2013 -- there are new emails from hassan. at least one of these email offers a path forward the "ICE DEPTH" measurements.
i think i may or may not have info about station coordinates and the likes.

these would be good updates. also, some met oracle views have dependencies on the tables (such as the ice depth) which would prevent us from refreshing.

let's revisit this before the end of october.

==================================================================================================
==================================================================================================
======================================================================================================
April 2013 -- general revision.

COmmonwealth COHM --- Ice surface change (like Taylor, VAnda & Vida)
Fryxell FRLM -- 2012 particle count?
F6 Snowfence FRLS --- mark data not collected, etc.

=====================================================================================
Most recent (mid-Nov) updates. I placed the 11/12 season data for "new" stations. I am done with
Garwood Valley, Mt Fleming and Friis Hills. For those, metadata is created and also, all the necessary tables, etc.
Perhaps I will review some sections of the metadata - make sure the entities order is correct, and make sure the
methods match - try to strip additional metadata from task list files and other info from Sue/Chris/Thomas. Panoramas
would be nice.

The process for a new station is involved. First, to meet database constraints, we need to make entries in several
tables "DATASET_TOC" MET_CORE_TOC. and METLOCS. One table for measurement (set of measurements). SOmetimes, post data insertion process is needed (wind). Finally, there is the discussion of the co-located stations. the arrays are a different date-time. see whether we can preserve original acronym.

more later.

I got the most recent 2011/12 met data from Thomas, yet I have a number of questions I sent, and more are cropping out. I will copy here the questions sent and also what I see.

2010/11
BHRM-airt-3m non existing (last 11/19/2010)
-radn - same
-rh - same
-soilt (0,5,10)- same
-surfchange - same
-wind - same

2011/12
BENM - done!! Note: most of season was not captured as station was off. beacon will be off as of 2013.

BOYM - done!!

BOYS (out of the game) bonney snowfence, discontinued and repurposed on 2006.

CAAM - in progress - a bit messy.
AIRT
things changed a bit in this station - Thomas says
" When I rebuilt the Canada Met Station, I moved the air temp/RH to the standard height of 3m. Before it was 2m. The 2m height no longer exists. That goes as well for the ice temps, except for the IceSurfTemp. "
So I added the AIRT3M, etc, before ingesting data.
Airt3m (aspirated. confirm is not the same w/thomas.). Update Watch this as compared with Particle Count.
AIRT3ASP (done)
ICET is confusing -- has icet for 50,1 and 25cm heights and "new icet" fir 25,50,75 and 100..... well. in addition, no values beyond 1/17/2009 (08/09 season)!
PC -- what is this? an empty table.. Update - Particle Count - Usually "Sensit" counter. Sometime Kinetic Energy. Need units for the later.
PRESSTA -- not collected (Done!)
RADN -- up to 1/26/2011 -- (10/11) seems like swrain, swradout, thmir, netrad)
imported season: concatenated sradin comm and sradoutcomm, not sure whether there is something doing about thmir netrad.
RH -- added RH3M column, imported season data (Done)
SURFTEMP -- this corresponds to IceSurfTemp. Imported as is - DONE
WIND -- done

COHM-
AIRT done
RH done
SURFTEMP done
WIND done
RADN done
ICET done

ICESURFCHANGE (data ends in 11/2009) This probably maps to "Ultrasonic Height". CHECK HERE
what do we do about "icesurftemp" CHECK HERE

EXEM
AIRT (Gap from 1/11/2010 to 11/9/2010) CHECK
PPT Done until 11/17/2011
RADN
RH
SOILT
WIND

FRLM
most data in database. somehow, i missed this file on 2012, 'found' it on april 2013.

all i need is check everything is in. perhaps the SURF Change?

FRLS
no data since end of 11/2009
HODM
last data point at 1/13/2011 - looks good.
AIRT done
ICET done
RH
RADN
WIND
ICESURFCHANGE
all done
HOEM
AIRT
RH
RADN
SOILT
WIND
PRESSTA
PPT
but, not Sonic Height.. Check!
TARM
AIRT
RH
RADN
ICESURF
WIND
no depth, no icet.
Not sure what to do with Sonicheight. Check!
UDHM
this has been discontinued for a long time
VAAM
missing data for 2010/11?
last date 11/29/2010
VIAM
missing data for 2010/11?
last date 11/19/2010
******************** im here, going alphabetically on met stations.**************************

CODES:

These are replacements you need to do:

R Out of Range.
Z Negative values zeroed out.
F Bad Value - Value below zeroing value.
B Bad Value - Value is equal to -6999 or known to be questionable
S is greater than a % of SwRadIn.
T Bad Value - Value below zeroing value.
N Wdir and WDirStD zeroed out because WSpd=0.
M Value Missing.

CORRESPONDENCE

======12/14/2012=================

I thought I would begin by addressing the first couple of your questions. I have cc:ed Thomas and Andrew on here as some of these questions pertain to how the data will be organized and accessed. I have numbered and left the asterisks next to your questions and placed ">>" at the beginning of my answers.

1.*****Rover stations data (Canada and Taylor Glacier) Do we have any data for the rovers ?

>>We do have data for the rover stations. Thomas indicated that he had previously submitted the data to the data manager, but I think that predates you. I have located the data on our network but it will take some digging to sort through before I can send to you.

2.*****Some newly added stations seem to be online, but I do not see all data Associated with them. For example the “Task List for 2010/11 says this About BYSM: [omitted] Remove station 01/21/2011 2046 and move instruments to South Side LH. Now, the last note is intriguing. Does BYSM cease to exist on Jan/2011 and it is now at SSLH? Do we still put the data up, and if we do, where is it?

>> The sensit station "BYSM" (Lake Bonney Sensit) ceases to exist when it is removed. The existing station at SSLH (South Shore Lake Hoare) gains a new parameters (columns) for the sensit sensor. There is no sensit data from the site yet because it was just downloaded this past season and has not been processed.

3.*****I have similar questions for other sensit data and Theta Soil stations. Basically, it boils down to this
1) where are those data
2) complete history of these stations, with complete metadata (start, end date, lat/lon, photo, sensors attached)

>> I think this question will best be answered by going station by station as you suggested. I think the big picture question is how to handle all these short term temporary experimental data. One suggestion would be to go in a similar manner as the "Steam Experiments (Short-term)" organized by type of station (sensit, snowfence, theta, etc) and location. What adds to the confusion is that some sites evolve into other sites. For example the old snow fence sites evolved into met and sensit sites. Perhaps we need to pick an appropriate time to break one station into the next and add a note in the metadata that the station evolved into another experiment.

I will answer the remaining questions in a separate email as they are more specific about data files and parameters.

========================================================================================

==bonney_snowfence============

The last usable data was recorded on January 4, 2006 @ 14:00. There was some intermittent data recorded during the 2006-2007 season, but not much. The station was modified in late January 2007 to support another experiment
=======================

============================
--the 'new' stations (need metadata, and check with.. you! on whether to cluster some of the data under same station acronym. (more on this later, but seems that similar acronyms may be co-located stations that measure different things, while possibly using different loggers, it may make sense to cluster the data under the same stations. we can also keep those split, no problem -- whatever we think it makes more sense to all)

========================
Inigo,

This will take me a bit to explain, and will have to do it when I have had a chance to settle in a bit more. Some of this I wrote up last year when I submitted the data to Sue, but it appears those notes did not get to you. I will see if I can dig them up in the next day or two.

-Thomas

On 2012-10-15 11:37, Inigo San Gil wrote:
>
> Hi Thomas, Hassan -
>
> You alluded to the wormherder stations (being in the dark) -Me too-
>
> But I also noticed also a sleuth of stations that were not "online" before.
> I mean, I see a lot of new 4-letter acronyms that do not seem to correspond
> to any of the 15 Stations I see. Again, these 15 that are "familiar" to me are:
> BENM, BOYM, BOYS, BHRM, CAAM, COHM, EXEM
> FRLM, FRLS, HODM, HOEM, TARM, UHDM, VAAM, VIAM
>
> But these others I see in the data you sent me.. not so familiar:
> LHS2, LHS1,BRTS, F6TS, BVTS, BRSS, F6SS, HDTR
> BVSS, EXSM, LHS, FRSM, F6MM, BRMM GADM.
>
> I think it is unfair to ask you to break this down for me, so
> I read a bit to make some sense of these, my source is the
> Met_Measurement_Summary PDF file you included.
>
> 1) Im going to guess LHS, LHS1, LHS2 (and HTRD??) are the same
> station at Lake Hoare South Side.
> Lake Hoare Soil (South Shore LH) (LHS1 and LHS2)
> Lake Hoare TDR (HTDR)
>
> We can host all these data in a consolidated LHSS, unless you
> prefer to keep them all four separated. Im going to need
> some metadata, perhaps I can fish it from some other project
> (like Soils)
>
> 2) Bonney Riegel Met (Formerly Bonney Snowfence station [BOYS] )
> (BRMM and BRSM) & Bonney Riegel Soil (BRSS and BRTS)
>
> Can we then insert these data (BRMM, BRSM, BRSS and BRTS) into
> that BOYS table, or do you want that into a new new table, like
> BRST -Bonney Riegel Met and Soil?
>
> It sounds to me that if it is in the same location, we should keep it
> in BOYS, but if it is sufficiently different, we start a new table.
>
> 3) More Snowfences. F6 Met/Sensit (Formerly F6 Snowfence) (F6MM and F6SM)
> F6 Soil (F6SS and F6TS). Again, should we put these in FRLS, or
> start a new one, like F6TS, and consolidate these 4 streams in one?
> Im assuming these four "F6" are co-located
>
> 4) Explorers Cove Met (EXEM and EXSM) Can I consolidate these two under
> EXEM ?
>
> 5) These look like new to the database, arent they?
>
> Friis Hills Met (FRSM)
> Mt. Fleming Met (FLMM)
> Garwood Valley Met (GADM)
>
> Also, Is there a way to get Lat/Longs, descriptions, some other metadata (when did these start, etc)
> for the new or renewed stations (including photos)?
>
> Thanks for clarifications, we can talk on the phone if that is more convenient for you.
>
> I will have some more questions, pertaining to mismatches on variables measured, such
> as the email I just sent about CAAM the Canada Glacier Met. Station. I compiled a spreasheet
> of all stations and all parameters measured that are currently in the database, and would like
> to ask you about missmatches and opportunites to consolidate a bit variables. More on this
> in a separate email.
>
> THANKS for your help!!
> Inigo
>
>
>

Other metadata:

(metVariables.xls)

ID NAME STATION AIRT RH WIND RADN SOILT PPT SNOW SNOWHT AIRPAR PAR* SOILPAR SURFCHANGE SOILSURFCHANGE AIRTASP ICET PC SURFTEMP ICESURF BLKR SURF PRESSTA SNOWFALL_MM DEPTH ONYXT
1 Beacon Valley BENM AIRT3M RH WDIR PAR SOILT0CM
AIRTMAX RH_COMMENTS WDIRSTD SWRADIN SOILT5CM
AIRTMIN WSPD SWRADOUT SOILT10CM
AIRT_COMMENTS WSPDMAX UVA SOILT_COMMENTS
WSPDMIN UVB
WIND_COMMENTS NETRAD
RADN_COMMENTS
2 Bonney BOYM AIRT3M RH WDIR PAR SOILT0CM PPT SNOWHT
AIRTMAX RH_COMMENTS WDIRSTD SWRADIN SOILT5CM PPT_COMMENTS SNOWHT_COMMENTS
AIRTMIN WSPD SWRADOUT SOILT10CM
AIRT_COMMENTS WSPDMAX LWRADIN SOILT_COMMENTS
WSPDMIN LWRADOUT
WIND_COMMENTS LWRADIN2
LWRADOUT2
RADN_COMMENTS
3 Bonney Snowfence BOYS AIRT1_3M PAR_AIR_PLUS1_5M PAR_SOIL_PLUS3_6M SURFCHANGE_CM
AIRT_COMMENTS AIRPAR_COMMENTS PAR_SOIL_PLUS1_8M SOILSURFCHANGE_COMMENTS
SOILPAR_COMMENTS
4 Brownworth BHRM AIRT3M RH WDIR PAR SOILT0CM SURFCHANGE_CM
AIRTMAX RH_COMMENTS WDIRSTD SWRADIN SOILT5CM SOILSURFCHANGE_COMMENTS
AIRTMIN WSPD SWRADOUT SOILT10CM
AIRT_COMMENTS WSPDMAX UVA SOILT_COMMENTS
WSPDMIN UVB
WIND_COMMENTS NETRAD
RADN_COMMENTS
5 Canada CAAM AIRT2M RH2M WDIR SWRADIN AIT3ASP ICET50CM PC SURFTEMP PRESSTA
AIRT1M RH1M WDIRSTD SWRADOUT AIRT3ASP_COMMENTS ICET1M PC_COMMENTS SURFTEMP_COMMENTS PRESSTA_COMMENTS
AIRT_COMMENTS RH_COMMENTS WSPD THMIR NEWICET_25CM
WSPDMAX NETRAD NEWICET_50CM
WSPDMIN RADN_COMMENTS NEWICET_75CM
WIND_COMMENTS NEWICET_100CM
ICET_COMMENTS
6 Commonwealth COHM AIRT3M RH WDIR SWRADIN SURFCHANGE_CM ICET20CM SURFTEMP
AIRTMAX RH_COMMENTS WDIRSTD SWRADOUT SOILSURFCHANGE_COMMENTS ICET1M SURFTEMP_COMMENTS
AIRTMIN WSPD LWRADIN ICET50CM
AIRT_COMMENTS WSPDMAX LWRADOUT ICETELEVATION
WSPDMIN LWRADIN2 ICET_COMMENTS
WIND_COMMENTS LWRADOUT2
THMIR
RADN_COMMENTS
7 Explorer's Cove EXEM AIRT3M RH WDIR PAR SOILT0CM PPT
AIRTDEL3M1 RH_COMMENTS WDIRSTD SWRADIN SOILT5CM PPT_COMMENTS
AIRT_COMMENTS WSPD SWRADOUT SOILT10CM
WSPDMAX RADN_COMMENTS SOILT_COMMENTS
WSPDMIN
WIND_COMMENTS
8 Fryxell FRLM AIRT3M RH WDIR PAR SOILT0CM RICHNO SURFELEVCHANGE
AIRTMAX RH_COMMENTS WDIRSTD SWRADIN SOILT5CM BLKR_COMMENTS SURF_COMMENTS
AIRTMIN WSPD SWRADOUT SOILT10CM
AIRTDEL3M1 WSPDMAX RADN_COMMENTS SOILT_COMMENTS
AIRT_COMMENTS WSPDMIN
WIND_COMMENTS
9 F6 Snowfence FRLS AIRT_3M SOILTN4_5M SNOWHT PAR_SOIL_P3_8M
AIRT_COMMENTS SOILTP1M SNOWHT_COMMENTS PAR_SOIL_P1_9M
SOILTP1_9M PAR_AIR_P1_5M
SOILTP3_8M SOILPAR_COMMENTS
SOILT_COMMENTS
10 Howard HODM AIRT1M RH WDIR SWRADIN ICET20CM
AIRT3M RH1M WDIRSTD SWRADOUT ICET1M
AIRTMAX RH_COMMENTS WSPD LWRADIN ICET_COMMENTS
AIRTMIN WSPDMAX RADN_COMMENTS
AIRT_COMMENTS WSPDMIN
WIND_COMMENTS
11 Hoare HOEM AIRT3M RH WDIR PAR SOILT0CM PPT PRESSTA SNOWFALL_M+U20M
AIRTMAX RH_COMMENTS WDIRSTD SWRADIN SOILT5CM PPT_COMMENTS PRESSTA_COMMENTS SNOWFALL_MM_COMMENTS
AIRTMIN WSPD SWRADOUT SOILT10CM
AIRTDEL3M1 WSPDMAX RADN_COMMENTS SOILT_COMMENTS
AIRT_COMMENTS WSPDMIN
WIND_COMMENTS
12 Taylor Glacier TARM AIRT1M RH WDIR SWRADIN ICET20CM ICESURF DEPTH
AIRT3M RH1M WDIRSTD SWRADOUT ICET50CM ICESURF_COMMENTS DEPTH_COMMENTS
AIRTMAX RH_COMMENTS WSPD LWRADIN ICET1M
AIRTMIN WSPDMAX LWRADIN2 NEWICET_25CM
AIRT_COMMENTS WSPDMIN RADN_COMMENTS NEWICET_50CM
WIND_COMMENTS NEWICET_75CM
NEWICET_100CM
ICET_COMMENTS
13 Upper Howard UHDM AIRT3M RH WDIR SWRADIN SNOWT20CM ICET20CM
AIRT_COMMENTS RH_COMMENTS WDIRSTD SWRADOUT SNOWT40CM ICET1M
WSPD RADN_COMMENTS SNOWT_COMMENTS ICET_COMMENTS
WSPDMAX
WSPDMIN
WIND_COMMENTS
14 Vanda Lake VAAM AIRT3M RH WDIR PAR SOILT0CM SURF_CHANGE ONYXT
AIRTMAX RH_COMMENTS WDIRSTD SWRADIN SOILT5CM SURFCHANGE_COMMENTS ONYXT_COMMENTS
AIRTMIN WSPD SWRADOUT SOILT10CM
AIRT_COMMENTS WSPDMAX RADN_COMMENTS SOILT_COMMENTS
WSPDMIN
WIND_COMMENTS
15 Lake Vida VIAM AIRT3M RH WDIR PAR SOILT0CM SURF_CHANGE
AIRT_COMMENTS RH_COMMENTS WDIRSTD SWRADIN SOILT5CM SURFCHANGE_COMMENTS
WSPD SWRADOUT SOILT10CM
WSPDMAX RADN_COMMENTS SOILT_COMMENTS
WSPDMIN
WIND_COMMENTS

Status: 

Priority: 

High