92 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
ggman12	b2f6a751fa	Refactor concat_parquet_to_final.py to accept date as an argument and streamline file handling	2026-02-17 17:27:41 -05:00
ggman12	88b00c1cf6	add .snapshots to .gitignore	2026-02-17 17:27:28 -05:00
ggman12	b6bf915cec	Filter rows by date in compress_parquet_part function	2026-02-17 16:59:09 -05:00
ggman12	6306aade16	Make global NUMBER_PARTS. remove print.	2026-02-17 16:42:59 -05:00
ggman12	9c54d9f1e4	remove print	2026-02-17 16:40:25 -05:00
ggman12	e8707ab853	get rid of unused code	2026-02-17 16:18:26 -05:00
ggman12	ca5cb23a4d	use date folder	2026-02-17 16:16:46 -05:00
ggman12	121dccf26c	concat file	2026-02-17 16:16:41 -05:00
ggman12	db98b3021a	change to use path	2026-02-17 16:16:35 -05:00
ggman12	94cf50ac3a	make return consistant	2026-02-17 16:09:09 -05:00
ggman12	ac177e8025	output .csv too	2026-02-17 16:09:01 -05:00
ggman12	6b7068bc84	works	2026-02-17 15:51:20 -05:00
ggman12	70ec797535	works	2026-02-17 15:46:07 -05:00
ggman12	1afe2bed4e	remvoe code from src/adsb/process_icao_chunk.py	2026-02-17 15:42:45 -05:00
ggman12	d3c52266e5	fix tar corrruption	2026-02-17 15:42:35 -05:00
ggman12	c0dca14b83	remove unused code	2026-02-17 14:57:29 -05:00
ggman12	1fc4a94743	do only a single day instead of multiple	2026-02-17 14:23:30 -05:00
ggman12	f29abad52a	output to parted tar.gz	2026-02-17 14:10:01 -05:00
ggman12	6eb84a894b	add notebooks/whatever.ipynb to .gitignore	2026-02-17 12:48:59 -05:00
ggman12	0c81490513	make it single day	2026-02-17 12:48:28 -05:00
ggman12	11ed7e597d	delete unused code	2026-02-17 12:48:01 -05:00
ggman12	24c0fc970c	use exclusive end_date	2026-02-17 12:47:44 -05:00
ggman12	c12e855b5a	change from 7 days to 1	2026-02-16 20:36:20 -05:00
ggman12	b55690638c	feat: implement download and concatenate script for workflow artifacts	2026-02-16 20:34:22 -05:00
ggman12	dcee136f09	refactor: update historical-adsb script to use 15-day chunks and improve argument handling	2026-02-16 20:14:04 -05:00
ggman12	035748fc61	skip using base release in run_local.py	2026-02-16 18:26:53 -05:00
ggman12	13432068e6	src/adsb/run_local.py works	2026-02-16 17:45:31 -05:00
ggman12	9cb4c5045b	remove compression from github action	2026-02-16 17:41:15 -05:00
ggman12	343a391a3f	change default chunk_days from 7 to 3	2026-02-16 16:35:05 -05:00
ggman12	2bc45ff6a4	increase retry_delay to 5 minutes.	2026-02-16 15:35:02 -05:00
ggman12	03291d93a8	add scripts/run_historical_adsb_action.py	2026-02-16 14:54:25 -05:00
ggman12	5883b459ac	fix bug with no dupliacte icaos across days	2026-02-15 21:08:17 -05:00
ggman12	f8ba66375b	preserve time	2026-02-15 21:08:03 -05:00
ggman12	7a62faecef	sort by time in end	2026-02-15 20:33:06 -05:00
ggman12	9964ce576b	slight update for compress by day	2026-02-15 20:32:33 -05:00
ggman12	be33fd2eaf	compress by day	2026-02-15 19:59:50 -05:00
ggman12	2b2095700f	use chunks in run_local	2026-02-15 19:53:09 -05:00
ggman12	a8b2b66952	fix .csv to .csv.gz transition openairframes-2026-02-17-develop openairframes-2026-02-17-main openairframes-2026-02-16-main openairframes-2026-02-16-develop	2026-02-15 19:08:51 -05:00
ggman12	3f38263a0c	stop depue that destroys previous days	2026-02-15 17:55:16 -05:00
ggman12	1a553d5f44	use date of file instead of min timestamp	2026-02-15 16:44:09 -05:00
ggman12	405855c566	deal with whole schema	2026-02-15 16:43:00 -05:00
ggman12	4e81dde201	fix date parsing	2026-02-15 14:55:32 -05:00
ggman12	fde8ef029c	update csv writing to handle empty data. Save space with higher gzip compression	2026-02-15 14:14:54 -05:00
ggman12	18ab51bd60	update naming	2026-02-15 13:45:03 -05:00
ggman12	83b9d2a76d	write gzip	2026-02-15 13:41:09 -05:00
ggman12	8874619ab0	write gzip	2026-02-15 13:41:02 -05:00
ggman12	823f291728	fix errors in daily release due to new .gz file	2026-02-15 13:21:51 -05:00
ggman12	982011b36f	end of year check	2026-02-14 22:42:32 -05:00
ggman12	1b15e43669	use .csv.gz	2026-02-14 22:22:14 -05:00
ggman12	f17adc4574	remvoe aws worker, reducer	2026-02-14 22:21:14 -05:00