首页 \ 问答 \ Talend ETL工具(Talend ETL tool)

Talend ETL工具(Talend ETL tool)

我正在开发一个迁移工具并使用Talend ETL工具(免费版)。

面临的挑战: -

是否可以创建每次运行时使用动态模式的Talend作业,即tMap组件中没有硬编码映射。

我希望用户提供输入CSV / Excel文件,作业应该根据该输入文件创建映射。 在talend有可能吗?

任何其他免费源ETL工具也可以是有用的,或任何样本作业。


I am developing a migration tool and using Talend ETL tool (Free edition).

Challenges faced:-

is it possible to create a Talend job that uses dynamic schema every time it runs i.e. no hard-coded mappings in tMap component.

I want user to give a input CSV/Excel file and the job should create mappings on the basis of that input file. Is it possible in talend?

Any other free source ETL tool can also be helpful, or any sample job.


原文:https://stackoverflow.com/questions/35498218
更新时间:2022-10-29 08:10

最满意答案

分享password (一个char[] )和salt (由SecureRandom选择的一个byte[] -8字节,使一个好的盐 - 不需要保密)与收件人的带外。 然后从这个信息中得出一个好的密钥:

/* Derive the key, given password and salt. */
SecretKeyFactory factory = SecretKeyFactory.getInstance("PBKDF2WithHmacSHA256");
KeySpec spec = new PBEKeySpec(password, salt, 65536, 256);
SecretKey tmp = factory.generateSecret(spec);
SecretKey secret = new SecretKeySpec(tmp.getEncoded(), "AES");

魔术数字(可以定义为常数)65536和256分别是关键派生迭代计数和密钥大小。

重要的推导函数被迭代以需要大量的计算工作,并且可以防止攻击者快速尝试许多不同的密码。 可以根据可用的计算资源来更改迭代计数。

密钥大小可以减少到128位,这仍然被认为是“强”加密,但是如果发现AES削弱的攻击,则不会给出很大的安全余地。

使用适当的块链模式,相同的派生密钥可用于加密许多消息。 在CBC中,对于每个消息生成随机初始化向量(IV),即使纯文本相同,也产生不同的密文。 CBC可能不是您可用的最安全的模式(参见下面的AEAD); 还有许多具有不同安全属性的其他模式,但它们都使用类似的随机输入。 在任何情况下,每个加密操作的输出是密文初始化向量:

/* Encrypt the message. */
Cipher cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");
cipher.init(Cipher.ENCRYPT_MODE, secret);
AlgorithmParameters params = cipher.getParameters();
byte[] iv = params.getParameterSpec(IvParameterSpec.class).getIV();
byte[] ciphertext = cipher.doFinal("Hello, World!".getBytes("UTF-8"));

存储ciphertextiv 。 在解密时, SecretKey以完全相同的方式重新生成,使用具有相同盐和迭代参数的密码。 使用此密钥初始化密码, 使用消息存储初始化向量:

/* Decrypt the message, given derived key and initialization vector. */
Cipher cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");
cipher.init(Cipher.DECRYPT_MODE, secret, new IvParameterSpec(iv));
String plaintext = new String(cipher.doFinal(ciphertext), "UTF-8");
System.out.println(plaintext);

Java 7包括对AEAD加密模式的API支持,OpenJDK和Oracle发行版中提供的“SunJCE”提供程序以Java 8开始实现。强烈建议其中一种模式代替CBC; 它将保护数据的完整性及其隐私。


带有消息“非法密钥大小或默认参数”的java.security.InvalidKeyException意味着密码强度有限; 无限制的权限策略文件不在正确的位置。 在JDK中,应该放在${jdk}/jre/lib/security

根据问题描述,看起来策略文件没有正确安装。 系统可以轻松拥有多个Java运行时; 仔细检查以确保使用正确的位置。


Share the password (a char[]) and salt (a byte[]—8 bytes selected by a SecureRandom makes a good salt—which doesn't need to be kept secret) with the recipient out-of-band. Then to derive a good key from this information:

/* Derive the key, given password and salt. */
SecretKeyFactory factory = SecretKeyFactory.getInstance("PBKDF2WithHmacSHA256");
KeySpec spec = new PBEKeySpec(password, salt, 65536, 256);
SecretKey tmp = factory.generateSecret(spec);
SecretKey secret = new SecretKeySpec(tmp.getEncoded(), "AES");

The magic numbers (which could be defined as constants somewhere) 65536 and 256 are the key derivation iteration count and the key size, respectively.

The key derivation function is iterated to require significant computational effort, and that prevents attackers from quickly trying many different passwords. The iteration count can be changed depending on the computing resources available.

The key size can be reduced to 128 bits, which is still considered "strong" encryption, but it doesn't give much of a safety margin if attacks are discovered that weaken AES.

Used with a proper block-chaining mode, the same derived key can be used to encrypt many messages. In Cipher Block Chaining (CBC), a random initialization vector (IV) is generated for each message, yielding different cipher text even if the plain text is identical. CBC may not be the most secure mode available to you (see AEAD below); there are many other modes with different security properties, but they all use a similar random input. In any case, the outputs of each encryption operation are the cipher text and the initialization vector:

/* Encrypt the message. */
Cipher cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");
cipher.init(Cipher.ENCRYPT_MODE, secret);
AlgorithmParameters params = cipher.getParameters();
byte[] iv = params.getParameterSpec(IvParameterSpec.class).getIV();
byte[] ciphertext = cipher.doFinal("Hello, World!".getBytes("UTF-8"));

Store the ciphertext and the iv. On decryption, the SecretKey is regenerated in exactly the same way, using using the password with the same salt and iteration parameters. Initialize the cipher with this key and the initialization vector stored with the message:

/* Decrypt the message, given derived key and initialization vector. */
Cipher cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");
cipher.init(Cipher.DECRYPT_MODE, secret, new IvParameterSpec(iv));
String plaintext = new String(cipher.doFinal(ciphertext), "UTF-8");
System.out.println(plaintext);

Java 7 included API support for AEAD cipher modes, and the "SunJCE" provider included with OpenJDK and Oracle distributions implements these beginning with Java 8. One of these modes is strongly recommended in place of CBC; it will protect the integrity of the data as well as their privacy.


A java.security.InvalidKeyException with the message "Illegal key size or default parameters" means that the cryptography strength is limited; the unlimited strength jurisdiction policy files are not in the correct location. In a JDK, they should be placed under ${jdk}/jre/lib/security

Based on the problem description, it sounds like the policy files are not correctly installed. Systems can easily have multiple Java runtimes; double-check to make sure that the correct location is being used.

相关问答

更多
  • 分享password (一个char[] )和salt (由SecureRandom选择的一个byte[] -8字节,使一个好的盐 - 不需要保密)与收件人的带外。 然后从这个信息中得出一个好的密钥: /* Derive the key, given password and salt. */ SecretKeyFactory factory = SecretKeyFactory.getInstance("PBKDF2WithHmacSHA256"); KeySpec spec = new PBEKeySp ...
  • 发现......他们使用ECB作为密码模式,因此没有IV。 我不会对此的“安全性”发表评论。 填充似乎是PKCS7(AES的默认值)。 密码“按原样”使用,只需用UTF8(或者甚至是ASCII)编码(因此它必须长32个字节)。 public static string EncryptText(string input, string password) { // Get the bytes of the string byte[] bytesToBeEncrypted = Encoding. ...
  • 由于密钥大小为256位的AES是我所需要的,所以我需要做的就是提供一个32字节的密钥。 而不是将字符串切成32个字符(这会改变键): $this->key = substr(hash('sha256', $key), 0, 32); 我应该把它打包成二进制字符串 $this->key = pack('H*', hash('sha256', $key)); Since AES with key size 256 bit is what I need, all I need to do is provide ...
  • 我最后联系了Jasypt的首席程序员DanielFernández和他的回答: 我担心Jasypt没有提供为SecretKeyFactory指定不同算法的方法以及Cipher本身的实例化。 抱歉。 我使用这段java代码来做这件事(没有Jasypt): public String encrypt(final String message) { final byte[] salt = generateSalt(); final Key key = createKey(salt); final C ...
  • 它似乎是256位散列的一部分,其周围有一些元数据。 我不熟悉用于加密的php库,但是知道你可以用php创建一个256bit的sha,并且应该有一个toHex()方法。 要检查密码,请使用sha256加密用户条目,并根据存储的值检查其十六进制值。 你有没看过它是否是沙的前1/4? 也许他们因为缺乏安全性而截断? 如果我有猜测,29可能是用户名,但可能是盐或什么的。 It appears to be part of a 256bit hash with some metadata around it. I'm ...
  • 我猜你的Java实现错过了密钥上的哈希步骤。 我正在使用密钥的SHA256哈希。 要测试c ++实现,请将代码更改为: QString encrypt(QByteArray r, const QString &password) { const char *sample = r.data(); string plain = password.toStdString(); string ciphertext; // Generate Cipher, Key, and CBC byte key[ AES::M ...
  • 您使用错误的密钥进行解密。 您使用密钥B加密,然后使用密钥A加密结果。然后,您尝试使用密钥B解密该结果,并使用密钥A解密最终结果。 每次解密时,都应指定用于执行“最新”加密操作的相同密钥。 所以: Encrypt Encrypt Decrypt Decrypt with B with A with A with B Plain text -> encrypted I -> enc ...
  • 根据crypto的Node.js 文档 : 建议开发人员使用crypto.pbkdf2()自己派生密钥和IV,并使用crypto.createCipheriv()创建Cipher对象。 用户不应在crypto.createCipher()中使用具有计数器模式的密码(例如CTR,GCM或CCM) 因此,您正在经历的行为本身并不是一个确定的行为。 我的猜测是底层密码对象是随机生成CTR模式的随机数,因此每次输出都不同。 要解决这个问题, crypto.createCipheriv按照文档建议使用crypto.c ...
  • 在Java中,您使用的是空/零:IV final IvParameterSpec iv = new IvParameterSpec(new byte[16]); 在C#中,您使用的是000102030405060708090A0B0C0D0E0F private static byte[] ivBytes = new byte[] { 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d ...
  • 默认情况下,Java仅支持128位加密。 如果您想超过该数量,则需要安装无限强度文件。 为此,下载jar并从zip中提取jar文件并将它们保存在$ {java.home} / jre / lib / security /中。 有关详细信息,请参阅此处: https : //stackoverflow.com/a/6481658/1008278 By default Java supports only 128-bit encryption. If you want to exceed more than t ...

相关文章

更多

最新问答

更多
  • 您如何使用git diff文件,并将其应用于同一存储库的副本的本地分支?(How do you take a git diff file, and apply it to a local branch that is a copy of the same repository?)
  • 将长浮点值剪切为2个小数点并复制到字符数组(Cut Long Float Value to 2 decimal points and copy to Character Array)
  • OctoberCMS侧边栏不呈现(OctoberCMS Sidebar not rendering)
  • 页面加载后对象是否有资格进行垃圾回收?(Are objects eligible for garbage collection after the page loads?)
  • codeigniter中的语言不能按预期工作(language in codeigniter doesn' t work as expected)
  • 在计算机拍照在哪里进入
  • 使用cin.get()从c ++中的输入流中丢弃不需要的字符(Using cin.get() to discard unwanted characters from the input stream in c++)
  • No for循环将在for循环中运行。(No for loop will run inside for loop. Testing for primes)
  • 单页应用程序:页面重新加载(Single Page Application: page reload)
  • 在循环中选择具有相似模式的列名称(Selecting Column Name With Similar Pattern in a Loop)
  • System.StackOverflow错误(System.StackOverflow error)
  • KnockoutJS未在嵌套模板上应用beforeRemove和afterAdd(KnockoutJS not applying beforeRemove and afterAdd on nested templates)
  • 散列包括方法和/或嵌套属性(Hash include methods and/or nested attributes)
  • android - 如何避免使用Samsung RFS文件系统延迟/冻结?(android - how to avoid lag/freezes with Samsung RFS filesystem?)
  • TensorFlow:基于索引列表创建新张量(TensorFlow: Create a new tensor based on list of indices)
  • 企业安全培训的各项内容
  • 错误:RPC失败;(error: RPC failed; curl transfer closed with outstanding read data remaining)
  • C#类名中允许哪些字符?(What characters are allowed in C# class name?)
  • NumPy:将int64值存储在np.array中并使用dtype float64并将其转换回整数是否安全?(NumPy: Is it safe to store an int64 value in an np.array with dtype float64 and later convert it back to integer?)
  • 注销后如何隐藏导航portlet?(How to hide navigation portlet after logout?)
  • 将多个行和可变行移动到列(moving multiple and variable rows to columns)
  • 提交表单时忽略基础href,而不使用Javascript(ignore base href when submitting form, without using Javascript)
  • 对setOnInfoWindowClickListener的意图(Intent on setOnInfoWindowClickListener)
  • Angular $资源不会改变方法(Angular $resource doesn't change method)
  • 在Angular 5中不是一个函数(is not a function in Angular 5)
  • 如何配置Composite C1以将.m和桌面作为同一站点提供服务(How to configure Composite C1 to serve .m and desktop as the same site)
  • 不适用:悬停在悬停时:在元素之前[复制](Don't apply :hover when hovering on :before element [duplicate])
  • 常见的python rpc和cli接口(Common python rpc and cli interface)
  • Mysql DB单个字段匹配多个其他字段(Mysql DB single field matching to multiple other fields)
  • 产品页面上的Magento Up出售对齐问题(Magento Up sell alignment issue on the products page)