github.com/wfusion/gofusion@v1.1.14/README.md

- English README version: you are reading now =。= translated by openai chat gpt4
- 中文 README: [README.md](assets/docs/zh/README.md)

# Usage Limitations

- golang: 1.18 - 1.22
- os: windows / darwin / linux
- arch: amd64 / arm64 / loong64

# Introduction

- Features: Highly configurable, highly extendable, high dependency replace ability, multi-component combinable, deeply
  integrated with dependency injection, facilitating efficient business development.
- Supported Components: db, http, i18n, lock, cache, log, mongo, redis, mq, routine, cron, async, metrics
- Special Features:
    - Supports YAML, JSON, TOML configuration file formats, highly configurable component parameters, with the ability
      to modify various component log switches at runtime.
    - Multiple db types supported: MySQL, Postgres, OpenGauss, SQLite, SQLServer, TiDB, ClickHouse, with nearly seamless
      dependency replacement.
    - Multiple mq types supported: RabbitMQ, Kafka, Pulsar, MySQL, Postgres, Redis, GoChannel, with seamless dependency
      replacement.
    - All components automatically register for dependency injection, aiding in building a system architecture based on
      dependency injection, utilizing uber/dig.
    - Distributed scheduled tasks, based on asynq.
    - Distributed asynchronous tasks, based on asynq.
    - HTTP services, scheduled tasks, asynchronous tasks, message queues all support router mode, with high freedom in
      registering function signatures.
    - Supports near and remote caching.
    - Multi-instance support for all components.
- Comparison
    - Compared to microservice frameworks like go-zero, go-micro, kratos, jupiter, kitex, dubbo-go, tarsgo, Gofusion is not a microservices framework, but is designed for dependency integration and combination, and can be
      used in conjunction with other frameworks.

# Quick Start

> Gofusion quick start

- Copy [test/config/configs/app.yml](test/config/configs/app.yml) to the `configs` directory in your business repository, or specify another location
  at startup with the `--config-file` parameter.
- Initialize gofusion with the following code:

```go
package main

import "github.com/wfusion/gofusion/config"

func main() {
	appSetting := &struct{}{} // Business configuration object
	defer config.Registry.Init(&appSetting)()
}
```

> fus quick start

- fus currently supports compression encryption encoding, random number generation, gorm-gen, asynq official open-source
  command line, watermill official open-source command line.

```bash
~ > go install github.com/wfusion/gofusion/common/fus@master
~ > fus -h
Gofusion CLI (v0.0.1 built with go1.18.10 darwin/arm64 from v1.1.4 on Sun Nov 12 18:05:03 CST 2023)

Capability:
  asynq client integrated
  watermill client with pubsub kafka, ampq, and io enabled integerate
  gorm gentool integerate
  encoder&decoder with cipher, compress, and print encoding
  random bytes generater

Usage:
  fus [command]

Available Commands:
  asynq       A CLI for asynq
  completion  Generate the autocompletion script for the specified shell
  dec         Decode data from stdin or filename
  enc         Encode data from stdin or filename
  gorm        A CLI for gorm gen-tool
  help        Help about any command
  mill        A CLI for watermill
  rnd         Generate cryptographically secure random bytes

Flags:
      --debug     print debug info
  -h, --help      help for fus
  -v, --version   version for fus

Use "fus [command] --help" for more information about a command.
```

# Unit Testing

- Unit test report generated based on allure, [test result](assets/unittest/unittest.html)
- Unit test coverage report generated based on github.com/nikolaydubina/go-cover-treemap, current coverage is 60%

![go-cover-treemap](assets/unittest/coverage.svg)

# Feature Summary

> Briefly describing gofusion capabilities

## Config

> Configuration Component, primarily provides gofusion initialization, configuration parsing, and graceful exit
> functionality for various components

- Business configuration class no longer needs to declare dependent configurations; all configurations are under
  the `base` node in the yml file.
- Components are automatically registered and gracefully exited in order based on business dependencies.
- Supports specifying configuration files based on environmental variables.
- Supports yaml, json, toml format configuration files (for example [app.yml](test/config/configs/app.yml)).
- Supports Command line args (for example [args.example](test/config/configs/full_commandline_args.example)).
- Supports multiple configuration priorities command line args > --config-file > app.local.yml > app.$env.yml > app.yml.
- Supports global debug, correspondingly enables debug mode for gorm, gin.
- Configurations support defining default values (`default: "{yaml format value}"`).
- Supports specifying multiple configuration files using --config-file.
- Supports auto reload of configuration content (mod time + sha256).
- Automatically calls flag.Parse, so no need for business to call it manually.
- Supports business objects implementing BeforeLoad and AfterLoad for callbacks before and after configuration loading.
- Business can customize initialization context, and manage the lifecycle and metrics looping of various components'
  internal implementations through this context.
- Supports encrypting passwords for db, mongo, redis through fus, and loading corresponding configurations from the
  configuration file in ciphertext.

## DB

> Database Component, provides relational database encapsulation features, some types are still under development
> testing...

- Supports multiple DB configurations.
- Based on gorm v2 supports mysql, postgres, opengauss, sqlite, sqlserver, tidb (untested), clickhouse (untested).
- Supports string-specified gorm log objects, default gorm log can dynamically adjust at runtime, log level, high
  consumption threshold.
- Supports nullable timestamp and bool soft delete fields, supports multiple soft delete field configurations in the
  same model, compatible with gorm.DeletedAt, compatible with sharding plugin.
- Combined with lock component to provide distributed lock capability.
- Encapsulated dal provides DalInterface interface, can use db.NewDAL directly, provides basic query, queryFirst,
  queryLast, count, pluck,
  take, insertOne, insertInBatch, save, update, updates, delete, firstOrCreate. For compatibility with other usages,
  provides ReadDB, WriteDB
  methods to return native -gorm.DB, provides IgnoreErr and CanIgnore encapsulation to help business avoid gorm no data
  return error when using first, last, pluck, task.
- Encapsulated db.Scan function supports full table scan.
- Encapsulated db.WithinTx function supports transactions.
- Encapsulated common class definition for model.
- Performance and latency dotting, currently no platform for reporting, temporarily unavailable.
- Gorm auto increment id issue in associated entities, currently can be configured according to different databases,
  declared in the configuration file through
  auto_increment_increment, if mariaDB or MySQL will query database auto-increment step by default according to sql show
  variables like 'auto_increment_increment'. Gorm issue see [Issue #5814](https://github.com/go-gorm/gorm/issues/5814)

> DB Sharding Plugin Features

mysql, postgres, sqlserver all passed tests.

- Supports multi-table sharding in one database, multi-column sharding in one table.
- Supports raw SQL.
- Supports AutoMigrate, DropTable.
- Multi-columns can use expression aggregation, default large digit shift according to binary stitching.
- Can shard table according to id column, default using snowflake algorithm, machine code is hash(host ip + local ip +
  pid) << 8 | last segment of local ipv4.
- Supports custom suffix name, default is original_table_name_column1_column2_0, suffix is 1, or 01, 001 depending on
  the number of sharded tables.
- Supports default table creation when creating, if the corresponding sharded table is not created, it will be created
  automatically, convenient for data migration.
- Combined with DalInterface interface, its batch insert, Save, Delete all support automatic sharding, that is, entities
  of different tables can be processed.

## HTTP

> HTTP component, provides HTTP component and error code encapsulation features

- Supports function signatures other than gin.HandlerFunc, automatically parses parameters from param, query, body based
  on HTTP request content-type, and analyzes returned data based on returned error.
- Supports various middlewares: cors cross-domain, original logic; logging for recording desensitized request logs; xss
  defense; trace id propagation, enhanced original logic; recover for exception capture, original logic; string-based
  middleware customization supported, yet this feature is temporarily unavailable.
- Error message texts support i18n.
- Error message texts support parameters (based on official golang text/template).
- Supports custom responses, use http.Response directly or inherit http.Embed for customization, mainly for
  compatibility with various business usages.
- Supports zero-copy gin.HandlerFunc based on gin framework, supports static filenames, supports io.ReadSeeker data
  streams.

## I18n

> Internationalization component, provides internationalization text features

- Supports multiple languages.
- Supports defining variables.
- Supports duplicate key detection, i.e., startup will fail if error code definitions are duplicated.

## Context

> Context component, provides field passing functionality based on context

- Supports initialization from gin based on HTTP component, compatible with past usages.
- user id, trace id propagation.
- In async component, supports user id, trace id, context.deadline propagation.
- In db component, supports gorm transaction propagation.
- In log component, supports automatic parsing and printing of user id, trace id information carried by context.

## Lock

> Distributed Lock component, provides distributed lock functionality

- Supports distributed locks based on Redis Lua, which are reentrant.
- Supports distributed locks based on Redis SETNX managed with a timeout.
- Supports distributed locks based on MySQL/MariaDB GET_LOCK/RELEASE_LOCK.
- Supports distributed locks based on a MongoDB collection and unique key, which are reentrant.
- Encapsulates lock.Within for distributed lock invocation.

## Cache

> Cache component, provides data caching functionality

- Supports near and remote caching, near-end uses github.com/bluele/gcache, remote currently only supports redis.
- Near-end cache supports arc, lfu, lru, simple eviction policies, arc used by default.
- Supports json, gob, msgpack, ctor for raw object encoding.
- Supports zstd, zlib, s2, gzip, deflate for compressed cache data post-serialization.
- Supports expiration time settings at the granularity of keys.
- Different encoding and compression algorithm read compatibility, i.e., modifying encoding and compression algorithms
  does not affect the reading and parsing of historical data.

## Log

> Log component, provides log printing and output functionality

- Based on go.uber.org/zap, compatible with past usage.
- Supports log.Fields as parameters directly brought into log printing.
- Supports console and file output, configuration similar to before, mainly for compatibility with the past, console
  color output configurable.
- Based on gopkg.in/natefinch/lumberjack.v2, supports automatic archiving of output files, defines maximum size, split
  cycle, maximum number, gzip compression for archived logs.
- Supports printing real file location of log call records (not a fixed stack depth skipping solution, as gorm, redis,
  mongo custom log printing is not at a fixed stack depth).
- Encapsulates TimeElapsed to record execution duration.

## Mongo

> Mongo component, provides mongo functionality

- Not yet encapsulated as an interface, nor provided dal encapsulation similar to db.
- Supports custom log printing, filters out printable mongo commands.
- Supports multiple mongo configurations.

## Redis

> Redis component, provides redis functionality

- Supports custom log printing, filters out non-printable redis commands.
- Supports multiple redis configurations.
- Supports combination with lock component, provides distributed lock capability.
- Supports combination with cache component, provides caching capability.
- Supports combination with cron component, provides scheduled task capability.
- Supports combination with async component, provides asynchronous task capability.

## Message Queue

> MQ component, provides message queue functionality

- Based on github.com/ThreeDotsLabs/watermill, forked and modified for compatibility with Go1.21, adjustments made for
  each pubsub open-source implementation.
- Supports amqp, rabbitmq, gochannel, kafka, pulsar, redis, mysql, postgres, all types passed all unit tests with
  consumer groups configured.
- Supports both pub/sub and pub/router modes, both modes can be used simultaneously.
    - When using both modes with the same configuration, router and sub will compete for consumption with raw and
      default messages.
    - When using both modes with the same configuration, router and sub will consume event messages repeatedly.
    - When registering multiple routers with the same configuration, router and sub will compete for consumption with
      raw and default messages.
    - When registering multiple routers with the same configuration, using event will trigger panic as registration name
      must be event type name causing duplication.
- Messages support default, raw, event types for publish/subscribe, raw is original message without encapsulation,
  default and event are gofusion-encoded messages.
    - Publish/subscribe default messages
        - Publisher uses Messages(), Objects() options to call Publish.
        - Subscriber uses Subscribe.
        - Router uses HandlerFunc, watermill.NoPublishHandlerFunc, watermill.HandlerFunc, custom func(ctx
          context.Context, payload *SerializableObj) error function for registration.
    - Publish/subscribe raw messages
        - Publisher uses Messages() option to call PublishRaw.
        - Subscriber uses SubscribeRaw.
        - Router uses HandlerFunc, watermill.NoPublishHandlerFunc, watermill.HandlerFunc, custom func(ctx
          context.Context, payload *SerializableObj) error function for registration.
    - Publish/subscribe event messages
        - Publisher initializes with NewEventPublisher, NewEventPublisherDI, uses Events() option to call PublishEvent.
        - Subscriber initializes with NewEventSubscriber, NewEventSubscriberDI, calls SubscribeEvent.
        - Router registers with EventHandler or EventHandlerWithMsg closure.
- Consumer execution function can be customized to func(ctx context.Context, payload *SerializableObj) error.
- Producers can directly pass message structures, auto-completes serialization and compression based on configuration,
  only valid for default and event messages.
- Supports json, gob, msgpack, ctor for raw object serialization, only valid for default and event messages.
- Supports post-serialization compression using zstd, zlib, s2, gzip, deflate, only valid for default and event
  messages.
- Supports langs, user_id, trace_id, context.deadline downstream propagation.
- Supports printing of business uuid and native id of messages, not propagated downstream.
- Supports defining custom Event for production and consumption of events, or for distinguishing different messages
  within a single Topic for business distribution, TODO: message deduplication and outdated message discarding to be
  developed.

## Routine

> Coroutine component, provides coroutine management and coroutine pool functionality

- Provides global coroutine count management, defines maximum coroutine count and graceful exit for business services.
- Supports debug mode, changes routine.Go, routine.Promise, and Pool.Submit calls to synchronous calls for easier
  testing and unit test writing.
- Encapsulates routine.Go, routine.Goc for go func calls, allows graceful exit; Args option supports arbitrary function
  parameters, WaitGroup option supports Done call, Channel option supports receiving function returns.
- routine.Loop, routine.Loopc compatible for certain business scenarios that don't require graceful exit.
- Based on github.com/fanliao/go-promise, encapsulates routine.Promise, routine.WhenAll, routine.WhenAny for easy
  multi-coroutine calling and waiting, supports arbitrary parameters.
- Based on github.com/panjf2000/ants/v2, encapsulates Pool interface for coroutine pool usage, supports arbitrary
  parameters.

## Cron

> Distributed scheduled task component, provides distributed scheduled task scheduling and management functionality

- Currently only supports asynq-based redis mode.
- Single service instance supports multiple cron instances, distributed environment supports multiple triggers and
  workers, exhibiting stateless service characteristics when both are enabled.
- Supports lock component integration to solve the issue of multiple trigger task re-distribution (without using lock
  component, use native unique option to avoid issue).
- Scheduled task execution function can be customized to func(ctx context.Context, payload *JsonSerializable) error
  signature, payload will be automatically parsed in gofusion.
- Scheduled task execution will carry cron_task_name and trace_id information in context.
- Configuration file supports asynq timeout, payload (json format), retry, deadline feature configurations.
- Besides configuring scheduled tasks in the configuration file, businesses can define custom task loader to append
  scheduled tasks.

## Async

> Distributed asynchronous task component, provides distributed asynchronous task scheduling and management
> functionality

- Currently only supports asynq-based redis mode.
- Supports langs, user_id, trace_id, context.deadline downstream propagation.
- Single service instance supports multiple async instances, distributed environment supports multiple producers and
  consumers, exhibiting stateless service characteristics when both are enabled.
- Asynchronous execution function can have a custom function signature, satisfying func(ctx context.Context, **selected
  serialization algorithm-supported one or more structures**) error, other parameters will be automatically parsed in
  gofusion.

## Metrics

> Monitoring pinpoint

- Currently only supports Prometheus, developed based on github.com/hashicorp/go-metrics, supports Prometheus's
  histogram pinpoint.
- Supports pull and push modes.
- Supports custom label constants.
- Supports configuration of golang program's runtime pinpoint, each component's pinpoint as well as running program
  hostname, ip label appending.
- Business pinpoint call passes golang channel, channel size and concurrency processing efficiency can be configured to
  avoid affecting business performance or duration.
- Businesses can configure pinpoint, if pinpoint task channel is full, strategy is default to discard pinpoint,
  configurable timeout or without timeout options are available.

## Common

> Common tools, providing frequently used functions and wrappers

| Path            | Capability                                                                                                                                                                                                                                                                       |
|:----------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| constant        | Constants, includes: reflection types, common symbols, time format                                                                                                                                                                                                               |
| constraint      | Generic constraints                                                                                                                                                                                                                                                              |
| di              | Dependency Injection based on uber dig                                                                                                                                                                                                                                           |
| env             | Retrieves runtime environment information                                                                                                                                                                                                                                        |
| fus             | Gofusion CLI, usage details in binary, install: go install github.com/wfusion/gofusion/common/fus@master                                                                                                                                                                         |
| infra           | Infrastructure, currently includes: mongo driver, common relational database drivers, redis driver, asynq asynchronous and scheduled task framework, watermill event-driven framework                                                                                            |
| utils/cipher    | Encryption/Decryption, supports des, 3des, aes, sm4, rc4, chacha20poly1305, xchacha20poly1305 encryption algorithms, ebc, cbc, cfb, ctr, ofb, gcm encryption modes, supports streaming processing                                                                                |
| utils/clone     | Supports deep copy of any type, based on github.com/huandu/go-clone                                                                                                                                                                                                              |
| utils/cmp       | Supports comparison of common types                                                                                                                                                                                                                                              |
| utils/compress  | Compression, supports zstd, zlib, s2, gzip, deflate compression algorithms, supports streaming processing, based on github.com/klauspost/compress                                                                                                                                |
| utils/serialize | Serialization, supports gob, json, msgpack, cbor serialization algorithms                                                                                                                                                                                                        |
| utils/sqlparser | SQL statement parsing, based on github.com/longbridgeapp/sqlparser and supports OFFSET FETCH statements                                                                                                                                                                          |
| utils/encode    | Printable encoding, supports hex, base32, base32-hex, base64, base64-url, base64-raw, base64-raw-url encoding algorithms, supports streaming processing, encryption, compression, and printable encoding combinations for []byte input processing and stream processing wrapping |
| utils/gomonkey  | Monkey patch, fork from github.com/agiledragon/gomonkey                                                                                                                                                                                                                          |
| utils/inspect   | Supports private field read/write, supports function and reflect.Type retrieval based on string, inspired by github.com/chenzhuoyu/go-inspect                                                                                                                                    |
| utils/pool.go   | Pooling generic encapsulation, by default provides pooled objects for bytes.Buffer and []byte                                                                                                                                                                                    |
| utils/xxx.go    | Other common functions or wrappers, covering compare, context, conv, enum, func, heap, ip, json, map, number, options, random, reflect, sets, slice, sort, string, time                                                                                                          |

# Thanks for

> The repository we forked 

- [hibiken/asynq](https://github.com/hibiken/asynq)
- [ThreeDotsLabs/watermill](https://github.com/ThreeDotsLabs/watermill)
- [hashicorp/go-metrics](https://github.com/hashicorp/go-metrics)
- [huandu/go-clone](https://github.com/huandu/go-clone)
- [chenzhuoyu/go-inspect](https://github.com/chenzhuoyu/go-inspect)
- [agiledragon/gomonkey](https://github.com/agiledragon/gomonkey)
- [fvbock/endless](https://github.com/fvbock/endless)
- [longbridgeapp/sqlparser](https://github.com/longbridgeapp/sqlparser)
- [jinzhu/configor](https://github.com/jinzhu/configor)
- [go-gorm/postgres](https://github.com/go-gorm/postgres)
- [natefinch/lumberjack](https://github.com/natefinch/lumberjack)
- [glebarez/sqlite](https://github.com/glebarez/sqlite)

# Todo List

- [ ] metrics each component
- [ ] mq component support event deduplication and outdated message discarding
- [ ] config component support Apollo dynamic config with viper lib
- [ ] http client wrapper for passing through trace id
- [ ] support rpc component
- [ ] support watermill type in cron and async components