Highly important missing features

Mean while I use my own prompt distiller - https://github.com/anfedoro/prompt_distil.git.. pretty much anough for me right now. Yo may try..Just it project indexing support Python only.. no justification/motivation to move to treesitter for other languages support.
depends on the model, reasoning and verbosity are configurable
Here is an example how it works

prompt-distil distill --text " amalyze the project and prepare Dataset implementration. Use pipeline API function  to access the parquet db or live data updates/tokenization. max_seq_len should be configurable. simple sample in the batch may merge few shorter records, longer record must be split on few items" --format markdown
✓ Validating input…
✓ Building/Using cache…
✓ Starting transcript distillation process…
✓ Protecting code identifiers…
✓ Invoking reconciliation model and performing symbol matching…
✓ Loading project symbol cache…
✓ Preparing distillation prompts…
  ✓ Cleaned backtick-enclosed words from prompt text
✓ Cleaning backtick symbols for distillation model…
  ✓ Received response from distillation model
  ✓ Parsed distillation model response into structured format
✓ Calling the distillation model for final prompt generation
✓ Rendering final prompts…
✓ Creating session summary…
✓ Printing results…
## Goal
Implement a Dataset implementation that uses the project's pipeline API to read data (from a parquet DB or a live data stream), tokenize records, and produce training batches with a 
configurable `max_seq_len` where short records may be merged into a single sample and long records are split into multiple items.

## Context
- Dataset should be part of the project's data layer (candidate file: lm/dataset/bgplm_data.py).
- Data sources: parquet DB or live streaming source accessed via the project's pipeline API.
- Tokenization must be applied (there is existing Tokenizer functionality in the project).
- `max_seq_len` must be configurable (applies to tokenized sequence length).
- Batch formation should allow merging several shorter records and splitting long records across multiple samples.
- Provide a simple example/sample in a batch demonstrating merging and splitting behavior.

## Change Request
**Required:**
- Use the project's pipeline API to access data from either the parquet DB or a live stream.
- Expose a configurable `max_seq_len` parameter that controls the maximum token length per sample.
- Apply tokenization to raw records before length-based merging/splitting.
- Merge multiple shorter records into a single batch sample when appropriate (sample-level packing) so long as the resulting token length does not exceed `max_seq_len`.
- Split records that exceed `max_seq_len` into multiple samples so that each resulting sample's token length <= `max_seq_len`.
- Include a simple example/sample in the delivered dataset (or tests) showing merged short records and a long record split into multiple items.

## Constraints
**Unclear requirements (handle carefully):**
- Exact name and signature of the pipeline API function(s) to call for parquet DB and live stream access (transcript references 'pipeline API function' but not a concrete identifier).
- Whether merging short records must preserve original record boundaries within the sample or can concatenate with separators.
- Exact splitting strategy for long records: whether splits should be at token boundaries, allow overlaps, or include special continuation tokens.
- Batching semantics: target batch size (number of samples), whether to pack across records to maximize tokens per sample, and whether shuffling is required.
- Schema/fields of parquet records (which fields to read and tokenize).
- Error handling and guarantees for live stream (e.g., retries, checkpointing, ordering).
- Whether padding/truncation/padding token behavior and special tokens (CLS/SEP) are required for samples shorter than `max_seq_len`.

## Acceptance Criteria
- A Dataset implementation (or module) exists that reads from parquet and from a live stream via the pipeline API.
- `max_seq_len` is configurable and respected: no output sample exceeds `max_seq_len` tokens.
- Tokenization is applied before length-based processing.
- Short records can be merged into single samples (packing) and at least one delivered example demonstrates merging of a few short records into a sample.
- Long records are split into multiple samples with each sample <= `max_seq_len`, and at least one delivered example demonstrates a long record being split.
- A minimal example or test is provided that exercises: parquet read path, live stream path (or a mocked stream), tokenization, merging behavior, and splitting behavior.

## Assumptions
- `max_seq_len` refers to tokenized sequence length (not raw character length).
- A pipeline API exists in the codebase and will provide either a parquet-reading iterator or a streaming iterator (the implementer will locate the concrete function names).
- An existing Tokenizer (or `tokenize` utility) is available in the project and should be used for tokenization.
- Merging short records means packing multiple tokenized records into one sample up to the `max_seq_len` limit (no requirement specified for separators or special tokens unless tests 
demand them).
- Splitting long records should produce contiguous token spans whose lengths do not exceed `max_seq_len` (details like overlap or continuation tokens are unspecified).
- The dataset implementation will live under the lm/ tree (suggested path: lm/dataset/bgplm_data.py) unless project conventions require a different location.

## Related (if any)
- lm/dataset/bgplm_data.py — `Dataset`
- lm/dataset/bgplm_data.py — `max_seq_len`

Hey there, thanks for the suggestions! Could you file each as a separate issue so we can track them better? Closing for now.