Met 2012-2013 season data process

Apr 13 --- index fix is done, resume other ops.

Apr 12 ---

Some indexes are corrupt due to data duplicates, im making some cleanup.
worked my way through some, but HOEM (all) and TARM (all) need work.
VERIFY that HOEM has a gap from Feb/13/12 to Sep/24/12

also, some materialized views need to be refreshed, mostly from FLMM onwards.

Apr 11 ---

almost done, need to run post-processing for means and avgs.

postprocessing done partly -- done SQLPLUS -> create_*.sql. Then refreshing materialized views (toad). Now done with COHM, do exem next.

1) Revisit / todo F6: F6SM,F6MM,F6TS - insist on metadata for sensits, etc.
2) MISM (Miers Valley)
3) Beacon relocation to Hoare?
4) Garwood?

Answers thus far ==> Garwood GADM was recycled, moved into MISM, Miers Valley. While we are at it, seems like GAFM will come online too.

Apr 7 ----
Finding the cruise velocity. I padded some gaps of older data, like some FRLM_PC files (though all counts of particles were zero !). i also uploaded a forgotten season for FRLM (2011) seemed I couldnt find the data last year (made a comment!).

Anyway, the process is quite simple. All proc.xlsx files were splitted, into files by measurement, corresponding to the tables. saved as CSV and XSLX, so they can be either uploaded through Toad or throuh the sqlloader. COmplex files like RADN are easier to handle in Toad, as it allows me to map columns, etc. Most of the files are handled well by the sqlloader. Here is the process for this sqldr.
-save as csv.
-match column headers exactly with dtabase.
-match file name with database table name.
-match order of columns too.
-run the perl script i made called csv2sqlldr "perl csv2sqlldr.pl CAAM_AIRT.csv (or CAAM_201213_AIRT.csv)
-except for airt and rh, edit the resulting sqlldr file and adjust headers slightly.
-as specail database user, in the unix shell, run sqlldr tiger/scott BINDSIZE=10485760 READSIZE=10485760 ROWS=10000 ERRORS=350 DIRECT=TRUE control=/path-to-file/BRHM_201213_WIND.csv.sqldr

What is the status?
Going by database order (as I see met data tables listed, from the top down)
-BENM (done)
-BOYM (done)
-boys disc.
-BRHM (done)
-CAAM (done)... conf. for hassan about less data, consolidate airtasp on airt?)
-COHM (done .. added a column to AIRT..)
-EXEM (done)
-EXSM (done, merge? expose?)
-FLMM (done)
-FRLM (done)
-FRSM (done)
-HODM (done)
-HOEM (done... where is the sonic height DB table?)
-TARM (done.. issues w/ IceSurfChange/DEPTH)
-VAAM (dome ICESURF again!) -- also, done last season excpt icesurf
-VIAM (done ICESURF issues too!) - also done last season ex. icesurf

-------------------------------------------------------------------------------
Im starting this on Apr 5 --- process, etc. Last year, I did more semimanually to get a good grasp of what was going on, study differences between tables, etc. This year I may automate somethings, but I am inclined to repeat the process until I understand fully the meteo data.

1 - extracted data from Dropbox to my server mcm\data\met\ folder.

---------------------------------------------------------------------------------

Here is from Hassan 3/12/2013 --

Met 201213
There are a total of 20 processed met files in csv format and 3 pdf
documents. The pdf documents are updated each year, these including
Measurement_Summary.pdf, Task_201213.pdf, and Interval_Dates.pdf.
Below are notes on the met data include:
1. Commonwealth Glacier station has two new columns for “AirTemp at 1
m”, and column for flagging.
2. Bonney Riegel Theta Soil Station and F6 Theta Soil Station was
previously submitted as two tables (soil temp and theta), I have
combined these into a single table.
3. Two stations were removed from service during this past season:
Beacon Valley and Explorers Cove Sensit Station.
4. No files are being submitted for station Bonney Riegel Sensit Met
(BRSM) as a programming error prevented the datalogger from recording
any data.
5. No files are available yet for Lake Hoare Theta Soil Station or
Lake Hoare TDR, these stations are problematic and will require more
time to process.

It would be good to revisit your questions on station metadata. I
think going through station by station would be the best approach.

-------------------------------------------

Status: 

Priority: 

Normal