Met - Glacier Season 13/14

-------------------------------update feb1 ---------------

a few met station related task remain, see below next update.

most glacier data entered -- unremarkable. perhaps 3 or 4 commonwealth related data files lacked header, matched by order, and visual verification that correspondences made sense (orders of magnitud, type of data, etc)

taylor mass change had errors from previous season --- time stamaps are 2020!. deleted records, reentered last years, and resorted, looks good now.

taylor snow denisty has sparse measurements. ask or double check.

========================

Update, Jan 29 2014=

-Fetched the zip file from Dropbox.
-Started data updates. First, Met data. Alphabetically by station name
-Process: Semi-automatic.
----Use excel to break out station data into sub-files of data. I.e, BOYM_AIRT,BOYM_RH, etc.
----Use excel to turn flags into actual meanings. Care to details when need to concatenate flags.
----Save Excel to CSV in subfolder "comma_delim". Play with "undo" to save the exact subset, but recover buffer to save time.
----Convert CSV to SQL Loader file. Use script csv2sqlldr.pl, on windoze machine. Some output files will need post-process, to include column names in the sqlldr header.
----Watch duplicates, there will be an overlap between data in DB and submitted data. Edit them out or use Toad w/ constraints enabled.
----Upload sqlldr to server, execute sqlloader command as ora user. Like this:

sqlldr usr/pwd BINDSIZE=10485760 READSIZE=10485760 ROWS=10000 ERRORS=350 DIRECT=TRUE control=/path/BOYM_201314_SNOWHT.csv.sqldr

----Use Toad to exec Update query on Data Set code. Like this:

UPDATE BOYM_X SET DATASET_CODE='BOYM_X' where DATE_TIME>=to_date('12/21/2012','MM/DD/YYYY')

----Repeat for each station subset, and each station. Takes a few days to complete.

-PROGRESS:

BOYM: Airt done, Rh done, Ppt done, Soilt done, Snowh done, Wind done, Rad done
BRHM: Airt done, Rh done, Radn done Soilt done, Surfchange done, wind done.
BRTS: here?
CAAM: Airt done, Rh done, Radn done, SurfTemp done, wind done.
COHM: Airt done, Rh done, icet done, icesurftemp done, Radn done, SurfTemp done, wind done.
FLMM: Airt done, Rh done, Pressta done, Wind done
FRLM: Airt done, Radn done, Rh done, Soilt done, surf elev change done, Wind done (PC, BRLK, others needed HERE)
FRSM: Airt done, Radn done, Rh done, Press done, Wind done
HODM: Airt done, Depth done, Rh done, Radn done, icet done, wind done
HOEM: airt done, rh done, radn done, SURF??, wind done, ppt done, pressta done, soilt done
GADM (mism) : airt and all done
TARM : all done
VAAM : problem of discontinuity
VIAM : all done except SURFCHANGE - discont.
****
*** need Vanda (gap). Also other newer stations (ho2m?). Some loose "Depth/surf" measurements.
***

**NEW Daily and Monthly stats for GADM/MISM and FRSM (Friis Hills) are already in place.** may need more for GAFM, E*, FLMM (fleming) others, but data for these are well, bad.

USeful for code conversion ---

CODES:

These are replacements you need to do:

R Out of Range.
Z Negative values zeroed out.
F Bad Value - Value below zeroing value.
B Bad Value - Value is equal to -6999 or known to be questionable
S is greater than a % of SwRadIn.
T Bad Value - Value below zeroing value.
N Wdir and WDirStD zeroed out because WSpd=0.
M Value Missing.

. . . . . . process to delete dupes . . . . . .
1- disable unique constraint on PK - DATE_TIME
2- drop index on DATE_TIME
3- remove duplicates, may use script if more than 10 or so.
3.a- sql snippet DELETE FROM TEMP_JOHN A WHERE ROWID > ( SELECT min(ROWID) FROM TEMP_JOHN B WHERE A.DATE_TIME = B.DATE_TIME);
4- reenable constraint
4.a may need to recr. index. likely will rebuild itself.

NOTES: Noted possible artifact on BOYM_AIRT between 12/22/2012 and 12/15/2012 (airt1M x airtMIN???)

=================Mid Jan 14============================

Hassan is leaving - care & attention to transition. Some notes from Hassan.
==========================================================
I have completed processing the met and glacier data from this past season. I posted the data to my dropbox account:

https://www.dropbox.com/s/axbsdenh0fl19lp/B504%20Data%20Submit%20201314.zip

I have cc:ed Peter Doran and Mike Gooseff to provide them with a sense of the data we submit each year. Glacier files Included are 24 csv files for Commonwealth, Howard, Canada, and Taylor glaciers, the measurement summary document (pdf), and an updated stake location map. Met files Included are 18 met csv files. Currently there are no files for Explorers Cove or Bonney Riegel met and sensit sites. Explorers Cove telemetry data contained too many data gaps to be processed and should be post processed when data is retrieved directly from the station storage module. I included a file for F6SM, but all are zeros. I recommended this station be removed next year.

Please let me know if you have questions when you begin to digest the data.
======================================
My glacier and met station responsibilities are being split between Peter Doran and Mike Gooseff. Doran's crew will take over the met maintenance and Gooseff the glacier side of things. I was able to train Doran and one of his crew, Kyle Cronin, this past November. Thomas Nylen will be training Gooseff and one of his assistants later this month. I will be transferring all hardcopies and digital archives and processing files to Doran and Gooseff later this month.

I am sorry to hear you are unable to head south this season. I hope next year is a possibility for you.
===================================================

Status: 

Priority: 

Normal