Blame: benthos.yaml - TykTechnologies/tyk

Tyk Open Source API Gateway written in Go, supporting REST, GraphQL, TCP and gRPC protocols

0 0 312 Go

[TT-12893]: Adding first implementation of streams API (#6511) ### User description <!-- Provide a general summary of your changes in the Title above --> ## Description Based off this POC [here](https://github.com/TykTechnologies/tyk/pull/6291) [TT-12893](https://tyktech.atlassian.net/browse/TT-12893) ## Related Issue <!-- This project only accepts pull requests related to open issues. --> <!-- If suggesting a new feature or change, please discuss it in an issue first. --> <!-- If fixing a bug, there should be an issue describing it with steps to reproduce. --> <!-- OSS: Please link to the issue here. Tyk: please create/link the JIRA ticket. --> ## Motivation and Context <!-- Why is this change required? What problem does it solve? --> ## How This Has Been Tested <!-- Please describe in detail how you tested your changes --> <!-- Include details of your testing environment, and the tests --> <!-- you ran to see how your change affects other areas of the code, etc. --> <!-- This information is helpful for reviewers and QA. --> ## Screenshots (if appropriate) ## Types of changes <!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: --> - [ ] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Refactoring or add test (improvements in base code or adds test coverage to functionality) ## Checklist <!-- Go over all the following points, and put an `x` in all the boxes that apply --> <!-- If there are no documentation updates required, mark the item as checked. --> <!-- Raise up any additional concerns not covered by the checklist. --> - [ ] I ensured that the documentation is up to date - [ ] I explained why this PR updates go.mod in detail with reasoning why it's required - [ ] I would like a code coverage CI quality gate exception and have explained why [TT-12893]: https://tyktech.atlassian.net/browse/TT-12893?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ ___ ### PR Type Enhancement, Tests ___ ### Description - Implemented a new `StreamingMiddleware` to handle streaming API functionality, including stream initialization and request processing. - Added comprehensive tests for streaming API scenarios, including single and multiple client interactions. - Developed a `PortalClient` for managing webhook credentials and interacting with the developer portal. - Introduced a `Stream` struct for managing stream configurations and handling unsafe components. - Enhanced the middleware interface to support unload functionality and integrated it into the API loader. - Updated the configuration to include streaming settings and adjusted dependencies to support new features. ___ ### Changes walkthrough 📝 <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Tests</strong></td><td><details><summary>3 files</summary><table> <tr> <td> <details> <summary><strong>mw_streaming_test.go</strong><dd><code>Add comprehensive tests for streaming API functionality</code>    </dd></summary> <hr> gateway/mw_streaming_test.go <li>Added tests for single and multiple client streaming API scenarios.<br> <li> Implemented setup functions for streaming API configurations.<br> <li> Included test cases for asynchronous API and WebSocket connections.<br> </details> </td> <td><a href="https://github.com/TykTechnologies/tyk/pull/6511/files#diff-a0d1bd0196a741537a3c850e340225c8993e49d709c838af0f1b48b9893af1da">+671/-0</a>  </td> </tr> <tr> <td> <details> <summary><strong>manager_test.go</strong><dd><code>Add unit tests for stream management functionality</code>              </dd></summary> <hr> internal/streaming/manager_test.go <li>Added tests for starting and stopping streams.<br> <li> Implemented tests for removing and whitelisting unsafe components.<br> <li> Verified stream configuration handling and error scenarios.<br> </details> </td> <td><a href="https://github.com/TykTechnologies/tyk/pull/6511/files#diff-c3ed576ae7d6430b9ac57b46c39d64981c274188c807ff4f0cfc4f15753e67af">+152/-0</a>  </td> </tr> <tr> <td> <details> <summary><strong>portal_test.go</strong><dd><code>Add tests for portal client webhook credential retrieval</code>  </dd></summary> <hr> internal/portal/portal_test.go <li>Added mock server for testing portal client interactions.<br> <li> Implemented tests for listing webhook credentials across multiple <br>apps.<br> <li> Verified correct handling of webhook data retrieval.<br> </details> </td> <td><a href="https://github.com/TykTechnologies/tyk/pull/6511/files#diff-2d5d9f02d0c6c149d531f5471b69936ccbf414a02d977813803fc3eafe15052d">+76/-0</a>    </td> </tr> </table></details></td></tr><tr><td><strong>Enhancement</strong></td><td><details><summary>6 files</summary><table> <tr> <td> <details> <summary><strong>mw_streaming.go</strong><dd><code>Implement streaming middleware for API gateway</code>                      </dd></summary> <hr> gateway/mw_streaming.go <li>Implemented <code>StreamingMiddleware</code> for handling streaming functionality.<br> <li> Added methods for initializing and managing streams.<br> <li> Introduced a mechanism to process requests and manage stream <br>lifecycles.<br> </details> </td> <td><a href="https://github.com/TykTechnologies/tyk/pull/6511/files#diff-6f565750150d990575c808f1ca8f38483160dc6edf05f1534cd0bedb27c2e6c8">+298/-0</a>  </td> </tr> <tr> <td> <details> <summary><strong>manager.go</strong><dd><code>Develop stream management and configuration handling</code>          </dd></summary> <hr> internal/streaming/manager.go <li>Created <code>Stream</code> struct for managing stream configurations.<br> <li> Added methods to start, stop, and reset streams.<br> <li> Implemented functionality to remove unsafe components from stream <br>configurations.<br> </details> </td> <td><a href="https://github.com/TykTechnologies/tyk/pull/6511/files#diff-8cda02479026085d17a6c5559a15bf59638174d16fb5f3ad65bebcecb25ad8d8">+236/-0</a>  </td> </tr> <tr> <td> <details> <summary><strong>portal_client.go</strong><dd><code>Add portal client for webhook credential management</code>            </dd></summary> <hr> internal/portal/portal_client.go <li>Introduced <code>PortalClient</code> for interacting with the developer portal.<br> <li> Added methods to list webhook credentials and fetch app details.<br> <li> Implemented client initialization with base URL and secret.<br> </details> </td> <td><a href="https://github.com/TykTechnologies/tyk/pull/6511/files#diff-0752a118f5d1513887f5ebf9930ae9c382a16798c3a4a301dac34523b5010289">+152/-0</a>  </td> </tr> <tr> <td> <details> <summary><strong>portal_output.go</strong><dd><code>Implement portal output for webhook integration</code>                    </dd></summary> <hr> internal/portal/portal_output.go <li>Implemented portal output for sending messages to webhooks.<br> <li> Added configuration for portal URL, secret, and event types.<br> <li> Developed methods for connecting and writing to webhooks.<br> </details> </td> <td><a href="https://github.com/TykTechnologies/tyk/pull/6511/files#diff-5225b7c3a3d3677eed2edb98be754e5cc70a124b0af26485511a0528f535048b">+128/-0</a>  </td> </tr> <tr> <td> <details> <summary><strong>api_loader.go</strong><dd><code>Integrate streaming middleware into API loader</code>                      </dd></summary> <hr> gateway/api_loader.go <li>Integrated <code>StreamingMiddleware</code> into the API loading process.<br> <li> Adjusted logic for unloading and reloading API specs.<br> </details> </td> <td><a href="https://github.com/TykTechnologies/tyk/pull/6511/files#diff-cdf0b7f176c9d18e1a314b78ddefc2cb3a94b3de66f1f360174692c915734c68">+9/-5</a>      </td> </tr> <tr> <td> <details> <summary><strong>middleware.go</strong><dd><code>Enhance middleware interface with unload capabilities</code>        </dd></summary> <hr> gateway/middleware.go <li>Enhanced middleware interface with unload functionality.<br> <li> Updated middleware creation to include unload hooks.<br> </details> </td> <td><a href="https://github.com/TykTechnologies/tyk/pull/6511/files#diff-703054910891a4db633eca0f42ed779d6b4fa75cd9b3aa4c503e681364201c1b">+15/-1</a>    </td> </tr> </table></details></td></tr><tr><td><strong>Configuration changes</strong></td><td><details><summary>1 files</summary><table> <tr> <td> <details> <summary><strong>config.go</strong><dd><code>Introduce streaming configuration in main config</code>                  </dd></summary> <hr> config/config.go <li>Added <code>StreamingConfig</code> struct to manage streaming settings.<br> <li> Integrated streaming configuration into the main config structure.<br> </details> </td> <td><a href="https://github.com/TykTechnologies/tyk/pull/6511/files#diff-fe44f09c4d5977b5f5eaea29170b6a0748819c9d02271746a20d81a5f3efca17">+26/-0</a>    </td> </tr> </table></details></td></tr><tr><td><strong>Dependencies</strong></td><td><details><summary>1 files</summary><table> <tr> <td> <details> <summary><strong>go.mod</strong><dd><code>Update dependencies for streaming and portal integration</code>  </dd></summary> <hr> go.mod <li>Updated dependencies to include streaming and portal-related packages.<br> <li> Adjusted versions for existing dependencies.<br> </details> </td> <td><a href="https://github.com/TykTechnologies/tyk/pull/6511/files#diff-33ef32bf6c23acb95f5902d7097b7a1d5128ca061167ec0716715b0b9eeaa5f6">+307/-23</a></td> </tr> </table></details></td></tr><tr><td><strong>Additional files (token-limit)</strong></td><td><details><summary>1 files</summary><table> <tr> <td> <details> <summary><strong>go.sum</strong><dd><code>...</code>                                                                                                            </dd></summary> <hr> go.sum ... </details> </td> <td><a href="https://github.com/TykTechnologies/tyk/pull/6511/files#diff-3295df7234525439d778f1b282d146a4f1ff6b415248aaac074e8042d9f42d63">+1134/-65</a></td> </tr> </table></details></td></tr></tr></tbody></table> ___ > 💡 PR-Agent usage: >Comment `/help` on the PR to get a list of all available PR-Agent tools and their descriptions --------- Co-authored-by: Leonid Bugaev <leonsbox@gmail.com> Co-authored-by: Martin Buhr <martin@tyk.io> Co-authored-by: Burak Sezer <burak.sezer.developer@gmail.com> 2024-09-24 17:30:08 +01:00			`input:`
			`stdin: {}`

			`output:`
			`http_server:`
			`address: "0.0.0.0:4196"`
			`path: /get`
			`stream_path: /get/stream`
			`ws_path: /get/ws`
			`allowed_verbs:`
			`- GET`