github.com/andrewrech/ih-abstract@v0.0.0-20210322142951-2fec1c8d0f38/docs.md

```
2021/01/26 13:06:29 starting

Select raw data for Immune Health report generation.

USAGE:

  < results-raw.csv | ih-abstract

DEFAULTS:

  -config string
    	Path to ih-abstract.yml SQL connection configuration file
  -no-filter
    	Save input data to .csv and exit without Immune Health filtering
  -old string
    	Path to existing results.csv output data from last run (optional)
  -print-config
    	Print an example configuration file and exit
  -sql
    	Read input from Microsoft SQL database instead of Stdin

DETAILS:

  ih-abstract streams input raw pathology results to the immune.health.report R package
  for report generation and quality assurance. The input is .csv data or direct streaming
  from a Microsoft SQL driver-compatible database. The output is filtered .csv/.txt files
  for incremental new report generation and quality assurance.

  Optionally, Immune Health filtering can be turned off to use ih-abstract as a general
  method to retrieve arbitrary or incremental pathology results.

  Quality assurance output consists of files containing
    1) unique, and
    2) new (never-before-seen)
  Immune Health report results for manual review.

  Dependencies are vendored and consist of the Go standard library and
  Microsoft SQL driver.

OUTPUT:

  Output for report generation:

    results.csv:                     all results
    results-increment.csv:           new results since last run
    new-ids.txt:                     patient identifiers with new results since last run

 Output for quality assurance:

    pdl1.csv:                        potential PD-L1 reports
    msi.csv:                         potential MSI reports
    cpd.csv:                         potential CPD reports
    wbc.csv:                         white blood cell counts

    pdl1-unique-strings.txt:         unique PD-L1 strings
    pdl1-unique-strings-new.txt:     unique PD-L1 strings, new vs. last run
    msi-unique-strings.txt:          unique MSI strings
    msi-unique-strings-new.txt:      unique MSI strings, new vs. last run

CONFIGURATION FILE:

  See 'ih-abstract --print-config'. Paths searched by default:

      $XDG_CONFIG_HOME/ih-abstract/ih-abstract.yml
      $HOME/.ih-abstract.yml
      ./ih-abstract.yml


TESTING:

 go test
                          NOTE: some integration tests require restricted
                          PHI-containing data. Data is available within our
                          organization upon request.

													NOTE: To test the live server connection, set
                          environment variable IH_ABSTRACT_TEST_CONFIG to
                          a test configuration file path.
                          These tests is disabled by default.

BENCHMARKING:

  go test -bench=.
                          NOTE: some benchmarks require restricted
                          organization VPN access. Access is available within our
                          organization upon request. To test the live server
                          connection, set environment variable
                          IH_ABSTRACT_TEST_CONFIG to a test configuration
                          file path. These tests is disabled by default.

```
<!-- Code generated by gomarkdoc. DO NOT EDIT -->

# ih\-abstract

```go
import "github.com/andrewrech/ih-abstract"
```

## Index

- [Constants](<#constants>)
- [func CPD(s string) bool](<#func-cpd>)
- [func Diff(oldFile *string, in chan []string, header []string) (out chan []string, done chan struct{})](<#func-diff>)
- [func DiffUnq(in chan []string, name string) (channels map[string](chan []string), done chan struct{})](<#func-diffunq>)
- [func Exclude(s string) bool](<#func-exclude>)
- [func MSI(s string) bool](<#func-msi>)
- [func New(r *Records, header []string, in chan []string, out chan []string, done chan struct{})](<#func-new>)
- [func PDL1(s string) bool](<#func-pdl1>)
- [func RecordID(header []string) (id string, err error)](<#func-recordid>)
- [func WbcLymph(s string) bool](<#func-wbclymph>)
- [func Whitespace(s []string) []string](<#func-whitespace>)
- [func Write(h []string, in map[string](chan []string)) (done chan struct{})](<#func-write>)
- [func WriteRows(in chan []string, name string, h []string, done chan struct{})](<#func-writerows>)
- [func connect(config string) (db *sql.DB, err error)](<#func-connect>)
- [func count(counter *int64, descr string, signal chan struct{})](<#func-count>)
- [func filterResults(in chan []string, header []string) (results map[string](chan []string), done chan struct{})](<#func-filterresults>)
- [func filterRow(l []string, colNames map[string]int, pat map[string](*regexp.Regexp), channels map[string](chan []string), counter *int64)](<#func-filterrow>)
- [func headerParse(h []string) (colNames map[string]int)](<#func-headerparse>)
- [func locateDefaultConfig() (config string, err error)](<#func-locatedefaultconfig>)
- [func main()](<#func-main>)
- [func mainInner(f flags, in *os.File)](<#func-maininner>)
- [func printConf()](<#func-printconf>)
- [func splitCh(in chan []string) (out1 chan []string, out2 chan []string, done chan struct{})](<#func-splitch>)
- [func usage()](<#func-usage>)
- [type Records](<#type-records>)
  - [func Existing(name *string) (rs *Records)](<#func-existing>)
  - [func prevUnq(f string) (r *Records)](<#func-prevunq>)
  - [func (r *Records) Add(l *[]string) (err error)](<#func-records-add>)
  - [func (r *Records) Check(l *[]string) (exists bool, err error)](<#func-records-check>)
- [type Store](<#type-store>)
- [type Writer](<#type-writer>)
  - [func File(name string, h []string) (w Writer)](<#func-file>)
- [type confVars](<#type-confvars>)
  - [func loadConfig(config string) (vars confVars, err error)](<#func-loadconfig>)
- [type flags](<#type-flags>)
  - [func flagParse() (f flags)](<#func-flagparse>)
- [type rawRecords](<#type-rawrecords>)
  - [func DB(config string, db *sql.DB) (r rawRecords)](<#func-db>)
  - [func read(f flags, in *os.File) (r rawRecords)](<#func-read>)
  - [func readCSV(in io.Reader) (r rawRecords)](<#func-readcsv>)
  - [func readSQLRows(rows *sql.Rows) (r rawRecords)](<#func-readsqlrows>)


## Constants

MsiReport is the string form of the regular expression used to match microsatellite instability reports of interest\.

```go
const MsiReport = "[Mm]icrosatellite[ ]+[Ii]nstability"
```

MsiResult is the string form of the regular expression used to extract microsatellite instability results\.

```go
const MsiResult = "[^\\.:]+findings[^\\.]+[Mm]icrosat[^\\.]+."
```

Pdl1Report is the string form of the regular expression used to match PD\-L1 reports of interest\.

```go
const Pdl1Report = "(?i)pd-?l1"
```

Pdl1Result is the string form of the regular expression used to extract PD\-L1 tumor/cancer score results\.

```go
const Pdl1Result = "(?i)(tumor proportion score|combined positive score \\(cps\\)|cps score):? ?[><~]* ?[0-9\\-\\.]+ ?%?"
```

SpacesAndBreaks is the string form of the replace\-all regular expression used to normalize whitespace in pathology report strings of interest\.

```go
const SpacesAndBreaks = `\s+`
```

## func [CPD](<https://github.com/andrewrech/ih-abstract/blob/main/filter.go#L38>)

```go
func CPD(s string) bool
```

CPD efficiently selects reports that are CPD reports using a lookup table\.

## func [Diff](<https://github.com/andrewrech/ih-abstract/blob/main/records.go#L193>)

```go
func Diff(oldFile *string, in chan []string, header []string) (out chan []string, done chan struct{})
```

Diff diffs old and new record sets\.

## func [DiffUnq](<https://github.com/andrewrech/ih-abstract/blob/main/unique.go#L27>)

```go
func DiffUnq(in chan []string, name string) (channels map[string](chan []string), done chan struct{})
```

DiffUnq identifies unique strings from an input stream and compares the unique strings to an existing output file\. The function returns 1\) unique strings and 2\) new strings compared to the existing output file\.

## func [Exclude](<https://github.com/andrewrech/ih-abstract/blob/main/filter.go#L82>)

```go
func Exclude(s string) bool
```

Exclude efficiently excludes unwanted report categories using a lookup table\.

## func [MSI](<https://github.com/andrewrech/ih-abstract/blob/main/filter.go#L66>)

```go
func MSI(s string) bool
```

MSI uses a lookup table to efficiently test if a string should be evaluated via regular expression as a potential PD\-L1 report\.

## func [New](<https://github.com/andrewrech/ih-abstract/blob/main/records.go#L121>)

```go
func New(r *Records, header []string, in chan []string, out chan []string, done chan struct{})
```

New identifies new Pathology database records based on a record hash\. For each new record\, the corresponding patient identifier to saved to a file\.

## func [PDL1](<https://github.com/andrewrech/ih-abstract/blob/main/filter.go#L48>)

```go
func PDL1(s string) bool
```

PDL1 uses a lookup table to efficiently test if a string should be evaluated via regular expression as a potential PD\-L1 report\.

## func [RecordID](<https://github.com/andrewrech/ih-abstract/blob/main/records.go#L176>)

```go
func RecordID(header []string) (id string, err error)
```

RecordID returns a single input data column name containing a person\-instance identifier\. The person instance identifier is either an MRN \(preferred\) or UID\.

## func [WbcLymph](<https://github.com/andrewrech/ih-abstract/blob/main/filter.go#L27>)

```go
func WbcLymph(s string) bool
```

WbcLymph efficiently selects records that are WbcLymph or lymphocyte counts using a lookup table\.

## func [Whitespace](<https://github.com/andrewrech/ih-abstract/blob/main/filter.go#L115>)

```go
func Whitespace(s []string) []string
```

Whitespace normalizes whitespace in report strings of interest\.

## func [Write](<https://github.com/andrewrech/ih-abstract/blob/main/write.go#L77>)

```go
func Write(h []string, in map[string](chan []string)) (done chan struct{})
```

Write writes results to output CSV files using a common header\.

## func [WriteRows](<https://github.com/andrewrech/ih-abstract/blob/main/write.go#L58>)

```go
func WriteRows(in chan []string, name string, h []string, done chan struct{})
```

WriteRows appends strings to a CSV file using a Writer\.

## func [connect](<https://github.com/andrewrech/ih-abstract/blob/main/connect.go#L11>)

```go
func connect(config string) (db *sql.DB, err error)
```

connect connects to an SQL database\.

## func [count](<https://github.com/andrewrech/ih-abstract/blob/main/utils.go#L11>)

```go
func count(counter *int64, descr string, signal chan struct{})
```

count counts processed lines per unit time\.

## func [filterResults](<https://github.com/andrewrech/ih-abstract/blob/main/filter.go#L169>)

```go
func filterResults(in chan []string, header []string) (results map[string](chan []string), done chan struct{})
```

filterResults filters a raw data input stream row by row\.

## func [filterRow](<https://github.com/andrewrech/ih-abstract/blob/main/filter.go#L126>)

```go
func filterRow(l []string, colNames map[string]int, pat map[string](*regexp.Regexp), channels map[string](chan []string), counter *int64)
```

filterRow filters a row of input data for matches to patterns of interest\.

## func [headerParse](<https://github.com/andrewrech/ih-abstract/blob/main/read.go#L151>)

```go
func headerParse(h []string) (colNames map[string]int)
```

headerParse parses input data column names\.

## func [locateDefaultConfig](<https://github.com/andrewrech/ih-abstract/blob/main/config.go#L37>)

```go
func locateDefaultConfig() (config string, err error)
```

locateDefaultConfig locates the configuration file in $XDG\_CONFIG\_HOME\, $HOME\, or the current directory\.

## func [main](<https://github.com/andrewrech/ih-abstract/blob/main/ih-abstract.go#L11>)

```go
func main()
```

ih\-abstract streams input raw pathology results to the immune\.health\.report R package for report generation and quality assurance\. The input is \.csv data or direct streaming from a Microsoft SQL driver\-compatible database\. The output is filtered \.csv files for incremental new report generation and quality assurance\. Optionally\, Immune Health filtering can be turned off to use ih\-abstract as a general method to retrieve arbitrary or incremental pathology results\.

## func [mainInner](<https://github.com/andrewrech/ih-abstract/blob/main/ih-abstract.go#L27>)

```go
func mainInner(f flags, in *os.File)
```

mainInner facilitates testing by allowing parameters to be passed to the main program code path\.

## func [printConf](<https://github.com/andrewrech/ih-abstract/blob/main/config.go#L14>)

```go
func printConf()
```

printConf prints an example SQL database configuration file

## func [splitCh](<https://github.com/andrewrech/ih-abstract/blob/main/utils.go#L37>)

```go
func splitCh(in chan []string) (out1 chan []string, out2 chan []string, done chan struct{})
```

splitCh splits a \[\]string channel into two channels\, sending results from the input channel onto both output channels

## func [usage](<https://github.com/andrewrech/ih-abstract/blob/main/cli.go#L38>)

```go
func usage()
```

usage prints usage\.

## type [Records](<https://github.com/andrewrech/ih-abstract/blob/main/records.go#L21-L24>)

Records provides thread safe access to Store\.

```go
type Records struct {
    Store
    sync.Mutex
}
```

### func [Existing](<https://github.com/andrewrech/ih-abstract/blob/main/records.go#L69>)

```go
func Existing(name *string) (rs *Records)
```

Existing creates a map of existing records\.

### func [prevUnq](<https://github.com/andrewrech/ih-abstract/blob/main/unique.go#L10>)

```go
func prevUnq(f string) (r *Records)
```

prevUnq adds previously identified unique strings from an existing output file to a hash map\.

### func \(\*Records\) [Add](<https://github.com/andrewrech/ih-abstract/blob/main/records.go#L27>)

```go
func (r *Records) Add(l *[]string) (err error)
```

Add adds a record\.

### func \(\*Records\) [Check](<https://github.com/andrewrech/ih-abstract/blob/main/records.go#L49>)

```go
func (r *Records) Check(l *[]string) (exists bool, err error)
```

Check checks that a record exists\.

## type [Store](<https://github.com/andrewrech/ih-abstract/blob/main/records.go#L18>)

Store is a blake2b hash map that stores string slices\.

```go
type Store map[[blake2b.Size256]byte](struct{})
```

## type [Writer](<https://github.com/andrewrech/ih-abstract/blob/main/write.go#L12-L18>)

Writer contains a file name\, connection\, CSV Writer\, and a 'done' signal to cleanup the connection\.

```go
type Writer struct {
    name    string
    conn    *os.File
    w       *csv.Writer
    counter int64
    done    func()
}
```

### func [File](<https://github.com/andrewrech/ih-abstract/blob/main/write.go#L21>)

```go
func File(name string, h []string) (w Writer)
```

File creates an output CSV write file\.

## type [confVars](<https://github.com/andrewrech/ih-abstract/blob/main/config.go#L27-L34>)

confVars is a struct of configuration variables required for the SQL database connection\.

```go
type confVars struct {
    Username string `yaml:"username"`
    Password string `yaml:"password"`
    Host     string `yaml:"host"`
    Port     string `yaml:"port"`
    Database string `yaml:"database"`
    Query    string `yaml:"query"`
}
```

### func [loadConfig](<https://github.com/andrewrech/ih-abstract/blob/main/config.go#L80>)

```go
func loadConfig(config string) (vars confVars, err error)
```

## type [flags](<https://github.com/andrewrech/ih-abstract/blob/main/cli.go#L10-L16>)

flagVars contains variables set by command line flags\.

```go
type flags struct {
    config   *string
    example  *bool
    noFilter *bool
    old      *string
    sql      *bool
}
```

### func [flagParse](<https://github.com/andrewrech/ih-abstract/blob/main/cli.go#L19>)

```go
func flagParse() (f flags)
```

flags parses command line flags\.

## type [rawRecords](<https://github.com/andrewrech/ih-abstract/blob/main/connect.go#L44-L48>)

rawRecords contains a header\, a channel of raw records\, and a channel indicating when raw records have been read\.

```go
type rawRecords struct {
    header []string
    out    chan []string
    done   chan struct{}
}
```

### func [DB](<https://github.com/andrewrech/ih-abstract/blob/main/connect.go#L51>)

```go
func DB(config string, db *sql.DB) (r rawRecords)
```

DB reads records from an Sql database\.

### func [read](<https://github.com/andrewrech/ih-abstract/blob/main/read.go#L15>)

```go
func read(f flags, in *os.File) (r rawRecords)
```

read reads raw input data\.

### func [readCSV](<https://github.com/andrewrech/ih-abstract/blob/main/read.go#L103>)

```go
func readCSV(in io.Reader) (r rawRecords)
```

readCSV reads records from a CSV file\.

### func [readSQLRows](<https://github.com/andrewrech/ih-abstract/blob/main/read.go#L40>)

```go
func readSQLRows(rows *sql.Rows) (r rawRecords)
```

readSQLRows reads rows of strings from an SQL database\.



Generated by [gomarkdoc](<https://github.com/princjef/gomarkdoc>)