Morph MORPH
SIGN IN SIGN UP
awsdocs / aws-doc-sdk-examples UNCLAIMED

Welcome to the AWS Code Examples Repository. This repo contains code examples used in the AWS documentation, AWS SDK Developer Guides, and more. For more information, see the Readme.md file below.

0 0 9 Java
Normal View History Raw
Run WRITEME over all Python services to generated READMEs. (#4603) * Run WRITEME over all Python services to generated READMEs. * Editorial feedback. * Add string to 40-char allow list.
2023-03-31 05:36:53 -07:00
# Amazon EMR code examples for the SDK for Python
Wrote two new EMR examples that show how to create short-lived and long-lived clusters. Cleaned up existing EMR examples that show how to run commands on running clusters. Added unit tests and documentation.
2020-08-07 17:39:33 -07:00
Run WRITEME over all Python services to generated READMEs. (#4603) * Run WRITEME over all Python services to generated READMEs. * Editorial feedback. * Add string to 40-char allow list.
2023-03-31 05:36:53 -07:00
## Overview
Wrote two new EMR examples that show how to create short-lived and long-lived clusters. Cleaned up existing EMR examples that show how to run commands on running clusters. Added unit tests and documentation.
2020-08-07 17:39:33 -07:00
Run WRITEME over all Python services to generated READMEs. (#4603) * Run WRITEME over all Python services to generated READMEs. * Editorial feedback. * Add string to 40-char allow list.
2023-03-31 05:36:53 -07:00
Shows how to use the AWS SDK for Python (Boto3) to work with Amazon EMR.
Wrote two new EMR examples that show how to create short-lived and long-lived clusters. Cleaned up existing EMR examples that show how to run commands on running clusters. Added unit tests and documentation.
2020-08-07 17:39:33 -07:00
Run WRITEME over all Python services to generated READMEs. (#4603) * Run WRITEME over all Python services to generated READMEs. * Editorial feedback. * Add string to 40-char allow list.
2023-03-31 05:36:53 -07:00
<!--custom.overview.start-->
<!--custom.overview.end-->
Wrote two new EMR examples that show how to create short-lived and long-lived clusters. Cleaned up existing EMR examples that show how to run commands on running clusters. Added unit tests and documentation.
2020-08-07 17:39:33 -07:00
Tools: WRITEME for python (#5845) Apply suggestions from code review Co-authored-by: Rachel Hagerman <110480692+rlhagerm@users.noreply.github.com> Apply suggestions from code review Co-authored-by: Rachel Hagerman <110480692+rlhagerm@users.noreply.github.com> Update python/example_code/s3/README.md Co-authored-by: Rachel Hagerman <110480692+rlhagerm@users.noreply.github.com> Added missing custom messages
2024-01-05 10:05:03 -05:00
_Amazon EMR is a web service that makes it easy to process vast amounts of data efficiently using Apache Hadoop and services offered by Amazon Web Services._
Wrote two new EMR examples that show how to create short-lived and long-lived clusters. Cleaned up existing EMR examples that show how to run commands on running clusters. Added unit tests and documentation.
2020-08-07 17:39:33 -07:00
Run WRITEME over all Python services to generated READMEs. (#4603) * Run WRITEME over all Python services to generated READMEs. * Editorial feedback. * Add string to 40-char allow list.
2023-03-31 05:36:53 -07:00
## ⚠ Important
Added snippet tags and SOS data for EMR examples. Update README to latest standards.
2021-09-22 16:13:37 -07:00
Tools: WRITEME for python (#5845) Apply suggestions from code review Co-authored-by: Rachel Hagerman <110480692+rlhagerm@users.noreply.github.com> Apply suggestions from code review Co-authored-by: Rachel Hagerman <110480692+rlhagerm@users.noreply.github.com> Update python/example_code/s3/README.md Co-authored-by: Rachel Hagerman <110480692+rlhagerm@users.noreply.github.com> Added missing custom messages
2024-01-05 10:05:03 -05:00
* Running this code might result in charges to your AWS account. For more details, see [AWS Pricing](https://aws.amazon.com/pricing/) and [Free Tier](https://aws.amazon.com/free/).
Run WRITEME over all Python services to generated READMEs. (#4603) * Run WRITEME over all Python services to generated READMEs. * Editorial feedback. * Add string to 40-char allow list.
2023-03-31 05:36:53 -07:00
* Running the tests might result in charges to your AWS account.
* We recommend that you grant your code least privilege. At most, grant only the minimum permissions required to perform the task. For more information, see [Grant least privilege](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#grant-least-privilege).
* This code is not tested in every AWS Region. For more information, see [AWS Regional Services](https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services).
Wrote two new EMR examples that show how to create short-lived and long-lived clusters. Cleaned up existing EMR examples that show how to run commands on running clusters. Added unit tests and documentation.
2020-08-07 17:39:33 -07:00
Run WRITEME over all Python services to generated READMEs. (#4603) * Run WRITEME over all Python services to generated READMEs. * Editorial feedback. * Add string to 40-char allow list.
2023-03-31 05:36:53 -07:00
<!--custom.important.start-->
<!--custom.important.end-->
Wrote two new EMR examples that show how to create short-lived and long-lived clusters. Cleaned up existing EMR examples that show how to run commands on running clusters. Added unit tests and documentation.
2020-08-07 17:39:33 -07:00
Run WRITEME over all Python services to generated READMEs. (#4603) * Run WRITEME over all Python services to generated READMEs. * Editorial feedback. * Add string to 40-char allow list.
2023-03-31 05:36:53 -07:00
## Code examples
Rearrange WRITEME to put prerequisites at the top of Code examples section (#4674) * Rearrange WRITEME to put prerequisites at the top of Code examples section. Run WRITEME over all Python services.
2023-04-13 11:38:14 -07:00
### Prerequisites
For prerequisites, see the [README](../../README.md#Prerequisites) in the `python` folder.
Install the packages required by these examples by running the following in a virtual environment:
```
python -m pip install -r requirements.txt
```
<!--custom.prerequisites.start-->
<!--custom.prerequisites.end-->
Run WRITEME over all Python services to generated READMEs. (#4603) * Run WRITEME over all Python services to generated READMEs. * Editorial feedback. * Add string to 40-char allow list.
2023-03-31 05:36:53 -07:00
### Single actions
Wrote two new EMR examples that show how to create short-lived and long-lived clusters. Cleaned up existing EMR examples that show how to run commands on running clusters. Added unit tests and documentation.
2020-08-07 17:39:33 -07:00
Run WRITEME over all Python services to generated READMEs. (#4603) * Run WRITEME over all Python services to generated READMEs. * Editorial feedback. * Add string to 40-char allow list.
2023-03-31 05:36:53 -07:00
Code excerpts that show you how to call individual service functions.
Wrote two new EMR examples that show how to create short-lived and long-lived clusters. Cleaned up existing EMR examples that show how to run commands on running clusters. Added unit tests and documentation.
2020-08-07 17:39:33 -07:00
Tools: Writeme canonical titles (#6382) * Fix some WRITEME issues and generate titles for Actions: * Open files with utf-8 encoding. * Case-insensitive README filename comparison. * Render titles for actions in READMEs. * Fixed WRITEME to check for combined language and version. * Remove case-insensitive README lookup, rename wrong case Readme.md to README.md.
2024-04-30 12:22:08 -07:00
- [AddJobFlowSteps](emr_basics.py#L148)
- [DescribeCluster](emr_basics.py#L105)
- [DescribeStep](emr_basics.py#L217)
- [ListSteps](emr_basics.py#L193)
- [RunJobFlow](emr_basics.py#L18)
- [TerminateJobFlows](emr_basics.py#L128)
Added snippet tags and SOS data for EMR examples. Update README to latest standards.
2021-09-22 16:13:37 -07:00
Run WRITEME over all Python services to generated READMEs. (#4603) * Run WRITEME over all Python services to generated READMEs. * Editorial feedback. * Add string to 40-char allow list.
2023-03-31 05:36:53 -07:00
### Scenarios
Added snippet tags and SOS data for EMR examples. Update README to latest standards.
2021-09-22 16:13:37 -07:00
Run WRITEME over all Python services to generated READMEs. (#4603) * Run WRITEME over all Python services to generated READMEs. * Editorial feedback. * Add string to 40-char allow list.
2023-03-31 05:36:53 -07:00
Code examples that show you how to accomplish a specific task by calling multiple
functions within the same service.
Added snippet tags and SOS data for EMR examples. Update README to latest standards.
2021-09-22 16:13:37 -07:00
Tools: writeme check for unmanaged folders logic remains wrong. (#6759)
2024-08-27 14:32:38 -04:00
- [Create a short-lived Amazon EMR cluster and run a step](../../example_code/emr)
Tools: WRITEME for python (#5845) Apply suggestions from code review Co-authored-by: Rachel Hagerman <110480692+rlhagerm@users.noreply.github.com> Apply suggestions from code review Co-authored-by: Rachel Hagerman <110480692+rlhagerm@users.noreply.github.com> Update python/example_code/s3/README.md Co-authored-by: Rachel Hagerman <110480692+rlhagerm@users.noreply.github.com> Added missing custom messages
2024-01-05 10:05:03 -05:00
- [Run a shell script to install libraries](install_libraries.py)
<!--custom.examples.start-->
<!--custom.examples.end-->
Wrote two new EMR examples that show how to create short-lived and long-lived clusters. Cleaned up existing EMR examples that show how to run commands on running clusters. Added unit tests and documentation.
2020-08-07 17:39:33 -07:00
Run WRITEME over all Python services to generated READMEs. (#4603) * Run WRITEME over all Python services to generated READMEs. * Editorial feedback. * Add string to 40-char allow list.
2023-03-31 05:36:53 -07:00
## Run the examples
Wrote two new EMR examples that show how to create short-lived and long-lived clusters. Cleaned up existing EMR examples that show how to run commands on running clusters. Added unit tests and documentation.
2020-08-07 17:39:33 -07:00
Run WRITEME over all Python services to generated READMEs. (#4603) * Run WRITEME over all Python services to generated READMEs. * Editorial feedback. * Add string to 40-char allow list.
2023-03-31 05:36:53 -07:00
### Instructions
Wrote two new EMR examples that show how to create short-lived and long-lived clusters. Cleaned up existing EMR examples that show how to run commands on running clusters. Added unit tests and documentation.
2020-08-07 17:39:33 -07:00
Run WRITEME over all Python services to generated READMEs. (#4603) * Run WRITEME over all Python services to generated READMEs. * Editorial feedback. * Add string to 40-char allow list.
2023-03-31 05:36:53 -07:00
<!--custom.instructions.start-->
<!--custom.instructions.end-->
Wrote two new EMR examples that show how to create short-lived and long-lived clusters. Cleaned up existing EMR examples that show how to run commands on running clusters. Added unit tests and documentation.
2020-08-07 17:39:33 -07:00
Python v3: Update READMEs in Python folder with WriteMe. (#5500) Updates to Python READMEs
2023-10-13 14:57:41 -05:00
Run WRITEME over all Python services to generated READMEs. (#4603) * Run WRITEME over all Python services to generated READMEs. * Editorial feedback. * Add string to 40-char allow list.
2023-03-31 05:36:53 -07:00
#### Create a short-lived Amazon EMR cluster and run a step
Wrote two new EMR examples that show how to create short-lived and long-lived clusters. Cleaned up existing EMR examples that show how to run commands on running clusters. Added unit tests and documentation.
2020-08-07 17:39:33 -07:00
Run WRITEME over all Python services to generated READMEs. (#4603) * Run WRITEME over all Python services to generated READMEs. * Editorial feedback. * Add string to 40-char allow list.
2023-03-31 05:36:53 -07:00
This example shows you how to create a short-lived Amazon EMR cluster that runs a step and automatically terminates after the step completes.
Wrote two new EMR examples that show how to create short-lived and long-lived clusters. Cleaned up existing EMR examples that show how to run commands on running clusters. Added unit tests and documentation.
2020-08-07 17:39:33 -07:00
Rearrange WRITEME to put prerequisites at the top of Code examples section (#4674) * Rearrange WRITEME to put prerequisites at the top of Code examples section. Run WRITEME over all Python services.
2023-04-13 11:38:14 -07:00
<!--custom.scenario_prereqs.emr_Scenario_ShortLivedEmrCluster.start-->
Run WRITEME over all Python services to generated READMEs. (#4603) * Run WRITEME over all Python services to generated READMEs. * Editorial feedback. * Add string to 40-char allow list.
2023-03-31 05:36:53 -07:00
Start the example by running the following at a command prompt:
Wrote two new EMR examples that show how to create short-lived and long-lived clusters. Cleaned up existing EMR examples that show how to run commands on running clusters. Added unit tests and documentation.
2020-08-07 17:39:33 -07:00
Tools: WRITEME for python (#5845) Apply suggestions from code review Co-authored-by: Rachel Hagerman <110480692+rlhagerm@users.noreply.github.com> Apply suggestions from code review Co-authored-by: Rachel Hagerman <110480692+rlhagerm@users.noreply.github.com> Update python/example_code/s3/README.md Co-authored-by: Rachel Hagerman <110480692+rlhagerm@users.noreply.github.com> Added missing custom messages
2024-01-05 10:05:03 -05:00
<!--custom.scenario_prereqs.emr_Scenario_ShortLivedEmrCluster.end-->
Run WRITEME over all Python services to generated READMEs. (#4603) * Run WRITEME over all Python services to generated READMEs. * Editorial feedback. * Add string to 40-char allow list.
2023-03-31 05:36:53 -07:00
Python v3: Update READMEs in Python folder with WriteMe. (#5500) Updates to Python READMEs
2023-10-13 14:57:41 -05:00
Run WRITEME over all Python services to generated READMEs. (#4603) * Run WRITEME over all Python services to generated READMEs. * Editorial feedback. * Add string to 40-char allow list.
2023-03-31 05:36:53 -07:00
<!--custom.scenarios.emr_Scenario_ShortLivedEmrCluster.start-->
Wrote two new EMR examples that show how to create short-lived and long-lived clusters. Cleaned up existing EMR examples that show how to run commands on running clusters. Added unit tests and documentation.
2020-08-07 17:39:33 -07:00
Shows how to write a job step that uses Apache Spark to estimate the value of pi by
performing a large number of parallelized calculations on cluster instances. Results
Run WRITEME over all Python services to generated READMEs. (#4603) * Run WRITEME over all Python services to generated READMEs. * Editorial feedback. * Add string to 40-char allow list.
2023-03-31 05:36:53 -07:00
are written to Amazon EMR logs and also to an S3 bucket.
<!--custom.scenarios.emr_Scenario_ShortLivedEmrCluster.end-->
Wrote two new EMR examples that show how to create short-lived and long-lived clusters. Cleaned up existing EMR examples that show how to run commands on running clusters. Added unit tests and documentation.
2020-08-07 17:39:33 -07:00
Run WRITEME over all Python services to generated READMEs. (#4603) * Run WRITEME over all Python services to generated READMEs. * Editorial feedback. * Add string to 40-char allow list.
2023-03-31 05:36:53 -07:00
#### Run a shell script to install libraries
Wrote two new EMR examples that show how to create short-lived and long-lived clusters. Cleaned up existing EMR examples that show how to run commands on running clusters. Added unit tests and documentation.
2020-08-07 17:39:33 -07:00
Run WRITEME over all Python services to generated READMEs. (#4603) * Run WRITEME over all Python services to generated READMEs. * Editorial feedback. * Add string to 40-char allow list.
2023-03-31 05:36:53 -07:00
This example shows you how to use AWS Systems Manager to run a shell script on Amazon EMR instances that installs additional libraries. This way, you can automate instance management instead of running commands manually through an SSH connection.
Rearrange WRITEME to put prerequisites at the top of Code examples section (#4674) * Rearrange WRITEME to put prerequisites at the top of Code examples section. Run WRITEME over all Python services.
2023-04-13 11:38:14 -07:00
<!--custom.scenario_prereqs.emr_Usage_InstallLibrariesWithSsm.start-->
<!--custom.scenario_prereqs.emr_Usage_InstallLibrariesWithSsm.end-->
Python v3: Update READMEs in Python folder with WriteMe. (#5500) Updates to Python READMEs
2023-10-13 14:57:41 -05:00
Start the example by running the following at a command prompt:
```
python install_libraries.py
```
Run WRITEME over all Python services to generated READMEs. (#4603) * Run WRITEME over all Python services to generated READMEs. * Editorial feedback. * Add string to 40-char allow list.
2023-03-31 05:36:53 -07:00
<!--custom.scenarios.emr_Usage_InstallLibrariesWithSsm.start-->
To install additional libraries on running cluster instances, run the following
command at a command prompt:
Wrote two new EMR examples that show how to create short-lived and long-lived clusters. Cleaned up existing EMR examples that show how to run commands on running clusters. Added unit tests and documentation.
2020-08-07 17:39:33 -07:00
Run WRITEME over all Python services to generated READMEs. (#4603) * Run WRITEME over all Python services to generated READMEs. * Editorial feedback. * Add string to 40-char allow list.
2023-03-31 05:36:53 -07:00
```
python install_libraries.py CLUSTER_ID SHELL_SCRIPT_PATH
```
This example is intended to be run as part of the tutorial in
[Installing Additional Kernels and Libraries](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-jupyterhub-install-kernels-libs.html).
The cluster specified by *CLUSTER_ID* must be set up to work with
Systems Manager. You must also have previously uploaded a shell script
to the Amazon S3 location specified by *SHELL_SCRIPT_PATH*.
<!--custom.scenarios.emr_Usage_InstallLibrariesWithSsm.end-->
Wrote two new EMR examples that show how to create short-lived and long-lived clusters. Cleaned up existing EMR examples that show how to run commands on running clusters. Added unit tests and documentation.
2020-08-07 17:39:33 -07:00
Run WRITEME over all Python services to generated READMEs. (#4603) * Run WRITEME over all Python services to generated READMEs. * Editorial feedback. * Add string to 40-char allow list.
2023-03-31 05:36:53 -07:00
### Tests
Wrote two new EMR examples that show how to create short-lived and long-lived clusters. Cleaned up existing EMR examples that show how to run commands on running clusters. Added unit tests and documentation.
2020-08-07 17:39:33 -07:00
Run WRITEME over all Python services to generated READMEs. (#4603) * Run WRITEME over all Python services to generated READMEs. * Editorial feedback. * Add string to 40-char allow list.
2023-03-31 05:36:53 -07:00
⚠ Running tests might result in charges to your AWS account.
Wrote two new EMR examples that show how to create short-lived and long-lived clusters. Cleaned up existing EMR examples that show how to run commands on running clusters. Added unit tests and documentation.
2020-08-07 17:39:33 -07:00
Run WRITEME over all Python services to generated READMEs. (#4603) * Run WRITEME over all Python services to generated READMEs. * Editorial feedback. * Add string to 40-char allow list.
2023-03-31 05:36:53 -07:00
To find instructions for running these tests, see the [README](../../README.md#Tests)
in the `python` folder.
Wrote two new EMR examples that show how to create short-lived and long-lived clusters. Cleaned up existing EMR examples that show how to run commands on running clusters. Added unit tests and documentation.
2020-08-07 17:39:33 -07:00
Run WRITEME over all Python services to generated READMEs. (#4603) * Run WRITEME over all Python services to generated READMEs. * Editorial feedback. * Add string to 40-char allow list.
2023-03-31 05:36:53 -07:00
<!--custom.tests.start-->
<!--custom.tests.end-->
## Additional resources
Wrote two new EMR examples that show how to create short-lived and long-lived clusters. Cleaned up existing EMR examples that show how to run commands on running clusters. Added unit tests and documentation.
2020-08-07 17:39:33 -07:00
Tools: WRITEME for python (#5845) Apply suggestions from code review Co-authored-by: Rachel Hagerman <110480692+rlhagerm@users.noreply.github.com> Apply suggestions from code review Co-authored-by: Rachel Hagerman <110480692+rlhagerm@users.noreply.github.com> Update python/example_code/s3/README.md Co-authored-by: Rachel Hagerman <110480692+rlhagerm@users.noreply.github.com> Added missing custom messages
2024-01-05 10:05:03 -05:00
- [Amazon EMR Management Guide](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-what-is-emr.html)
- [Amazon EMR API Reference](https://docs.aws.amazon.com/emr/latest/APIReference/Welcome.html)
- [SDK for Python Amazon EMR reference](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/emr.html)
Run WRITEME over all Python services to generated READMEs. (#4603) * Run WRITEME over all Python services to generated READMEs. * Editorial feedback. * Add string to 40-char allow list.
2023-03-31 05:36:53 -07:00
<!--custom.resources.start-->
<!--custom.resources.end-->
Wrote two new EMR examples that show how to create short-lived and long-lived clusters. Cleaned up existing EMR examples that show how to run commands on running clusters. Added unit tests and documentation.
2020-08-07 17:39:33 -07:00
---
Run WRITEME over all Python services to generated READMEs. (#4603) * Run WRITEME over all Python services to generated READMEs. * Editorial feedback. * Add string to 40-char allow list.
2023-03-31 05:36:53 -07:00
Wrote two new EMR examples that show how to create short-lived and long-lived clusters. Cleaned up existing EMR examples that show how to run commands on running clusters. Added unit tests and documentation.
2020-08-07 17:39:33 -07:00
Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
Run WRITEME over all Python services to generated READMEs. (#4603) * Run WRITEME over all Python services to generated READMEs. * Editorial feedback. * Add string to 40-char allow list.
2023-03-31 05:36:53 -07:00
SPDX-License-Identifier: Apache-2.0
Reference in New Issue Copy Permalink