S3keysensor

Posted onby

Bases: airflow.sensors.base_sensor_operator.BaseSensorOperator

S3keysensor

Waits for a key (a file-like instance on S3) to be present in a S3 bucket.S3 being a key/value it does not support folders. The path is just a keya resource.

Import S3keysensor

Sample Test with S3KeySensor. GitHub Gist: instantly share code, notes, and snippets. Feb 12, 2021 The Bucketkey in S3 is having a suffix (generated with timestamp), now how to call using in S3KeySensor. I don't want to specify as None, as I'm keeping exceptions as well. Also tried calling thru '.' (asterisk) at the end. But It's poking, but not linking to any path or file. Bases: airflow.sensors.basesensoroperator.BaseSensorOperator Waits for a key (a file-like instance on S3) to be present in a S3 bucket. S3 being a key/value it does not support folders. The path is just a key a resource. Jul 28, 2020 S3KeySensor: S3 Key sensors are used to wait for a specific file or directory to be available on an S3 bucket. Hooks are interfaces to services external to the Airflow Cluster.

S3keysensor Airflow Example

S3keysensor

S3keysensor Timeout

Parameters
  • bucket_key (str) – The key being waited on. Supports full s3:// style urlor relative path from root level.

  • bucket_name (str) – Name of the S3 bucket

  • wildcard_match (bool) – whether the bucket_key should be interpreted as aUnix wildcard pattern

  • aws_conn_id (str) – a reference to the s3 connection

  • verify (bool or str) –

    Whether or not to verify SSL certificates for S3 connection.By default SSL certificates are verified.You can provide the following values:

    • False: do not validate SSL certificates. SSL will still be used

      (unless use_ssl is False), but SSL certificates will not beverified.

    • path/to/cert/bundle.pem: A filename of the CA cert bundle to uses.

      You can specify this argument if you want to use a differentCA cert bundle than the one used by botocore.

template_fields = ['bucket_key', 'bucket_name'][source]
poke(self, context)[source]